+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201016/B1e --output-run DEEP/20201016/B1e/science#step1/20240313T213234Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 17.0108 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.9050 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.8730 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/pipeline_orig.qgraph DEEP/20201016/B1e/science#step1/20240313T213234Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.9031 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 19.4496 seconds; current memory usage: 0.377 Gibyte, delta: 0.199 Gibyte, peak delta: 0.187 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.368 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0264 seconds; current memory usage: 0.377 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.368 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201016_B1e_science#step1_20240313T213234Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 1.1330 seconds; current memory usage: 0.377 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.368 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 2.7958 seconds; current memory usage: 0.379 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.379 Gibyte (main), 0.368 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: 71367c66-b956-46c1-bae7-8ddd3c464d5d
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 23240215717632)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x15230defe250>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-13 21:34:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-13 21:34:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-13 21:34:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-13 21:34:10 proc_lsst.multi:170 [INFO]  [multi] job_id 12374
proc_lsst.multi INFO: [multi] job_id 12374
2024-03-13 21:34:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:10 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 12374
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 12374
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 12374
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x15230da2c090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash c4c49f5dcdb933fc05a86c31fc8b4b7c
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/802a8304-af35-4868-87ea-56e4a557449d_isr_946074_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/802a8304-af35-4868-87ea-56e4a557449d_isr_946074_54.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x15230df33110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x15230b25d390 state=pending>
parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x15230b25ebd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash cbe7b98f0f4dffe63c78d228196b2f6a
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80653471-3521-4223-a4b2-5ebfd10b8ec8_isr_946073_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80653471-3521-4223-a4b2-5ebfd10b8ec8_isr_946073_11.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x15230be06ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x15230bf3d9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x15230bf3d3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 3ff3ce0a99ae383275ef37244d2e3b80
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5c0a64a9-8a0a-43bc-a1d9-0bcd8ac0820a_isr_946074_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5c0a64a9-8a0a-43bc-a1d9-0bcd8ac0820a_isr_946074_20.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x15230bf3ca50 state=pending>
parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x15230b488910 state=pending>
parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x15230b488a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash bd647351e9efd02ebd9c49cf594ce7d6
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e1768d48-e88a-4b05-81a5-7b0b37d5d7a2_isr_946073_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e1768d48-e88a-4b05-81a5-7b0b37d5d7a2_isr_946073_39.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x15230af09a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x15230bc14850 state=pending>
parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x15230bc16d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 9d8c7107994466e70d7896ccdf68ecc9
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d069d79b-93f7-417d-ac26-6044ab198beb_isr_946180_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d069d79b-93f7-417d-ac26-6044ab198beb_isr_946180_60.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x15230bc17290 state=pending>
parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x15230bf2f350 state=pending>
parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x15230be91b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 67b8d71ad99cf1e705bec2b967b59928
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4ee107b8-0be2-4a74-8b4a-7d3edd0b1454_isr_946072_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4ee107b8-0be2-4a74-8b4a-7d3edd0b1454_isr_946072_60.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x15230be919d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x15230b119450 state=pending>
parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x15230ab94050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 81916b39c6c1de45a14b93c1404859af
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c0cdfdd6-0a35-4a7f-87f5-7303c702a3f1_isr_946180_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c0cdfdd6-0a35-4a7f-87f5-7303c702a3f1_isr_946180_7.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x15230ab96590 state=pending>
parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x15231083d050 state=pending>
parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x15231083fcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash f1a8a468f5c1b4da86aca6d0ce591102
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/14e1a92c-7ea6-4704-ba9b-5922941867f6_isr_946180_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/14e1a92c-7ea6-4704-ba9b-5922941867f6_isr_946180_25.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x15230a499890 state=pending>
parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x15230b211310 state=pending>
parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x15230b213850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 1d4ec301342e1d742f55c8187087d839
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2739ffb6-a372-4193-b835-337e4e57c37d_isr_946072_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2739ffb6-a372-4193-b835-337e4e57c37d_isr_946072_11.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x15230b2139d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x15230c8af690 state=pending>
parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x15230aeb56d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 8f5dd95028311b77f3a46714c178c911
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8dbb4db9-035f-4d90-9692-20b384b28b7e_isr_946074_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8dbb4db9-035f-4d90-9692-20b384b28b7e_isr_946074_37.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x15230beb42d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x15230aaba490 state=pending>
parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x15230aabad10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 554fd98c41f58d713222474fa7f9fb9f
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/af2c0791-f400-4ace-bf51-8dbd0b72898e_isr_946072_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/af2c0791-f400-4ace-bf51-8dbd0b72898e_isr_946072_44.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x15230a8419d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x15230b3b8410 state=pending>
parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x15230b3ba290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 384eab375465ed547f418045adef00d5
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f466c88e-5d4e-488b-8802-a69c2be083a5_isr_946182_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f466c88e-5d4e-488b-8802-a69c2be083a5_isr_946182_44.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x15230bd9d510 state=pending>
parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x15230bd9df90 state=pending>
parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x15230be0d990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 95da361f14be3f91158b4b695d1ea5b9
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/8ded0a21-2aa0-4c4e-a260-acc8298885a7_isr_946180_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/8ded0a21-2aa0-4c4e-a260-acc8298885a7_isr_946180_24.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x15230be0ee50 state=pending>
parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x15230d075490 state=pending>
parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x15230d075990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash ac6c3a5731b4deb0b00a6baae6a97b89
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/36aee1d1-7c58-4e1f-b922-da2934361737_isr_946074_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/36aee1d1-7c58-4e1f-b922-da2934361737_isr_946074_7.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x15230dfc68d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x15230c6c4510 state=pending>
parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x15230c6c5a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 926c54112ec1c9f0dac305a734055796
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8d2804a7-3fd2-4b7c-964d-d55bfb603451_isr_946072_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8d2804a7-3fd2-4b7c-964d-d55bfb603451_isr_946072_50.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x15230c6c6c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x15230a3ca790 state=pending>
parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x15230bccb610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash ee3157a937ee5bad04add3061cbb1cab
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ba443340-e3da-4ede-a5c8-4625efe00941_isr_946073_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ba443340-e3da-4ede-a5c8-4625efe00941_isr_946073_6.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x15230ad2e590 state=pending>
parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x152310748550 state=pending>
parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x152310749310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash d0603a519fa10c85115ad9b47a66c9db
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/07a677cb-555b-46f8-bb0e-c8dcbbda989d_isr_946180_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/07a677cb-555b-46f8-bb0e-c8dcbbda989d_isr_946180_30.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x152310748910 state=pending>
parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x15230b9a02d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x15230b9a17d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 22869fe627e93bedc97c2cf0c64c08cf
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f889a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d1c13ab8-f554-4301-9a1f-fbd98aeb7823_isr_946072_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d1c13ab8-f554-4301-9a1f-fbd98aeb7823_isr_946072_19.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x15230b9a1750 state=pending>
parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x15231084c3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x15231084e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 042869ada14de0386d9480d7f5126763
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b382c1d0-bb7e-4f35-afa1-6bf74e3a0694_isr_946181_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b382c1d0-bb7e-4f35-afa1-6bf74e3a0694_isr_946181_41.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x15231084eb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x15230b7db9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x15230a7cc790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash ceaf182d379b70fc9b13faa409c5ff0c
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/37f5f22c-6613-442a-b62f-49049d83d713_isr_946074_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/37f5f22c-6613-442a-b62f-49049d83d713_isr_946074_10.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x15230a6b8790 state=pending>
parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x15231b2a7250 state=pending>
parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x15230c0bb810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 4366b692210121ee387dad75f39fcd2c
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/68f8d5ce-3270-4af1-ad99-8011b4db20d3_isr_946182_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/68f8d5ce-3270-4af1-ad99-8011b4db20d3_isr_946182_19.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x15230c0ba950 state=pending>
parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x15230c0b9f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x15230bc88210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 78435dfef5d35e3dbb11b4605ee64759
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0b8c9127-68e7-443f-ab6b-87c5b1392e7b_isr_946182_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0b8c9127-68e7-443f-ab6b-87c5b1392e7b_isr_946182_61.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x15230c0bbf10 state=pending>
parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x152311081090 state=pending>
parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x152311082290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 4093ecf14fb1046829531897fc143a08
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/98370b0f-235d-4a24-878a-50b488624732_isr_946072_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/98370b0f-235d-4a24-878a-50b488624732_isr_946072_13.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x152311082dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x15230a57bc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x15230bb13150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 5a5f3d9f4046c6d0d0471d1a9ebc2daa
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/82938aa3-03d1-4a90-a7a0-5a3bbd25bbe4_isr_946073_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/82938aa3-03d1-4a90-a7a0-5a3bbd25bbe4_isr_946073_23.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x15230bb11c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x15230bb138d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x15230b3e84d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 1e821c9614100efdc1356a93f81ba969
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3ea720bd-9f87-4cf5-a411-7ee95de0d8da_isr_946181_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3ea720bd-9f87-4cf5-a411-7ee95de0d8da_isr_946181_58.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x15230be06f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x15230c7e8bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x15230c7eb490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 0aa172e1253163bab226a9468d5cfff9
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3a0e9159-e3ed-4b6d-92f2-437adc03151b_isr_946182_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3a0e9159-e3ed-4b6d-92f2-437adc03151b_isr_946182_28.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x15230aa40f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x15230af09950 state=pending>
parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x15230bfa56d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 725e7bd093833b70f0c516efa8aad1ba
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6739dd26-5da9-4aa9-b56a-51e08e261b3b_isr_946181_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6739dd26-5da9-4aa9-b56a-51e08e261b3b_isr_946181_36.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x15230bfa79d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x15230adef9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x15230adee390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash a950f1b358620eb7211df11d5f4d32d6
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/5544f9d8-7ee6-4cd5-9d06-d989eaa0c1e1_isr_946182_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/5544f9d8-7ee6-4cd5-9d06-d989eaa0c1e1_isr_946182_21.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x15230b6a5910 state=pending>
parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x15230b6a6b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x15230bdeef10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 4d1672d19872300e904d8f078917941e
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ef37a707-fb2a-4d71-9eea-6f5ee703ccdf_isr_946180_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ef37a707-fb2a-4d71-9eea-6f5ee703ccdf_isr_946180_61.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x15230bded550 state=pending>
parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x15230bc7f8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x15230bc7cdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 2f7f78fc82995ca55f01f7ce5dd88241
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d587ba4b-827a-4185-8ef2-2d9ac5708c47_isr_946073_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d587ba4b-827a-4185-8ef2-2d9ac5708c47_isr_946073_20.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x15230a48ae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x15230b5d2c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x15230c6f7c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 2263a942ec9f719a87402737541c5c23
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/fb69275d-2211-438f-abe9-2069b1fa0bf2_isr_946182_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/fb69275d-2211-438f-abe9-2069b1fa0bf2_isr_946182_51.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x15230c6f5390 state=pending>
parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x15230c6f4350 state=pending>
parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x15230b7cc510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 56f50180de5a7d78eb6a5056e2226112
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6ae940f9-4a5e-4872-892d-5ba16808bd0c_isr_946181_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6ae940f9-4a5e-4872-892d-5ba16808bd0c_isr_946181_14.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x15230b7cdf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x15230b7ce8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x15230c6d0bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 6485744568fe8d2ad7d0f402235350ed
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/0de60304-6635-40c6-b829-768f79591285_isr_946181_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/0de60304-6635-40c6-b829-768f79591285_isr_946181_55.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x15230c6d3f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x15230c6d12d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x15231aba4e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 0de16d9b862df72bbf7dddc40b84bbbd
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ff9b1499-6fb7-41fe-b2b8-d1803346188b_isr_946182_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ff9b1499-6fb7-41fe-b2b8-d1803346188b_isr_946182_50.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x15230c6d3550 state=pending>
parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x15231aba7a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x15230c6f99d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 06b3f8e99c47c1ec2ed0c23c7e1ef161
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/114f42e1-a86e-45a0-aadd-46ecf1680bb8_isr_946074_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/114f42e1-a86e-45a0-aadd-46ecf1680bb8_isr_946074_48.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x15230c6faa50 state=pending>
parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x15230c6f8b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x15230b094d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 5c15571b473c4fdc013a7cb3662235f9
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/40bffd92-6e31-4e85-bb79-dc4fcbb6d485_isr_946182_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/40bffd92-6e31-4e85-bb79-dc4fcbb6d485_isr_946182_43.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x15230b6d1c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x15230b097590 state=pending>
parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x15231106fa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash f50fc50737ae075e3f07a06220446bf8
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e8bb882b-5184-4bab-8f01-c679b5829b5b_isr_946073_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e8bb882b-5184-4bab-8f01-c679b5829b5b_isr_946073_48.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x15231106ff50 state=pending>
parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x15231106c8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x15231106c050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 085c1fad3b86743c97717affc8c40b40
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3dd11fc1-5698-4fc2-9f5b-4a044c4dd2d9_isr_946180_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3dd11fc1-5698-4fc2-9f5b-4a044c4dd2d9_isr_946180_48.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x15231106f050 state=pending>
parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x15231106cf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x15231106d3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 64c675779265eabc48b2de548e54b94c
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/58a3f5e7-6687-4199-ab63-2d236cb9541c_isr_946181_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/58a3f5e7-6687-4199-ab63-2d236cb9541c_isr_946181_46.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x15231106f2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x15230c6d8d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x15230c6d99d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 28c671455e2a8919d28ec606e6de2c0a
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5a144e4b-acb4-4cb8-a444-4d624385db32_isr_946074_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5a144e4b-acb4-4cb8-a444-4d624385db32_isr_946074_5.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x15230c6da790 state=pending>
parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x15230c6dab90 state=pending>
parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x15230c6dbcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash da7e39fa7b8d7c3f2b98d4d178381f53
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/61f3b29d-be37-4ba3-a3da-e31c76320bdc_isr_946074_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/61f3b29d-be37-4ba3-a3da-e31c76320bdc_isr_946074_4.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x15230c6d9e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x15230c6daf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x15230c757290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 70d9b4752836876b5d8674546816e0cd
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7dc1f173-7bf4-4acb-b239-03ba3e113a97_isr_946181_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7dc1f173-7bf4-4acb-b239-03ba3e113a97_isr_946181_61.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x15230c6da310 state=pending>
parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x15230c757dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x15230c757b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash c8a767ec427955db53d4b6cee4ba7ff3
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/145714e5-1525-4254-be8a-e403bc385e6b_isr_946073_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/145714e5-1525-4254-be8a-e403bc385e6b_isr_946073_18.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x15230c7576d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x15230c6d9b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x15230bcc13d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash f680fa1d39aa4ed2a8b0e957a71a2ca0
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2c004503-eb71-40d1-805b-764e737a6a89_isr_946181_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2c004503-eb71-40d1-805b-764e737a6a89_isr_946181_34.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x15230bcc2810 state=pending>
parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x15230ba5d590 state=pending>
parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x15230ba5fb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash a58305330994aac65f04dbafec20a16b
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3de6f9e0-8be0-47cf-a9b3-1fff75ee1295_isr_946182_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3de6f9e0-8be0-47cf-a9b3-1fff75ee1295_isr_946182_60.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x15230ba5c450 state=pending>
parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x15230bd80650 state=pending>
parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x15230bd812d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 90aea989930ebb9f700931da3b3fd777
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d11b15f3-4367-463d-832a-10f721b1270f_isr_946073_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d11b15f3-4367-463d-832a-10f721b1270f_isr_946073_54.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x15230bd81890 state=pending>
parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x15230a520210 state=pending>
parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x152310871650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash b4670e632229d121546828e055471933
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/214dcb40-cf1e-44f4-9643-0dd74ce7c983_isr_946072_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/214dcb40-cf1e-44f4-9643-0dd74ce7c983_isr_946072_40.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x152310870f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x1523108735d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x152310873d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash cf434908d966c0d72d5383bd41f4d8bd
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c15f73bc-ce98-4b5a-a369-9240d8481637_isr_946180_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c15f73bc-ce98-4b5a-a369-9240d8481637_isr_946180_19.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x152310872490 state=pending>
parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x15230bd81310 state=pending>
parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x15230ba64c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 256fbaee1104d1dcbbbfd25ae7fea0e8
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/af4a3b76-43ed-4341-8231-92bda52dd70c_isr_946074_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/af4a3b76-43ed-4341-8231-92bda52dd70c_isr_946074_31.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x15230ba64950 state=pending>
parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x1523109c8ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x1523109c8e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 185ca1dff4b0ffae291b748add9e4bf0
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/6fde5dc1-9890-4d9d-90ad-a2095b5fe440_isr_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/6fde5dc1-9890-4d9d-90ad-a2095b5fe440_isr_946074_61.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x1523109c96d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x1523109c8050 state=pending>
parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x1523109caf50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 85114afb7f590dadc2f1bd5dd8a42ea1
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9a3d01ff-13db-47bd-b97f-a8899027db36_isr_946072_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9a3d01ff-13db-47bd-b97f-a8899027db36_isr_946072_52.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x1523109cbfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x15230b75a250 state=pending>
parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x15230c06af90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash c6f03d1075bbeda6b526631b230cf677
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3d865830-82d8-4c48-a5c7-4cf5a468eefb_isr_946072_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3d865830-82d8-4c48-a5c7-4cf5a468eefb_isr_946072_12.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x15230c06bb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x15230c06a650 state=pending>
parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x15230c068cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 99f7ab28a31f55e90c9991585bfbd8f7
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce3b1349-906c-4743-af52-2c80f3b9af03_isr_946180_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce3b1349-906c-4743-af52-2c80f3b9af03_isr_946180_38.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x15230c06be10 state=pending>
parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x152310ea0d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x15230a108ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 6a3ace04ee8bfe91dbc239982855cfd2
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/21c01109-d822-4f8f-9c15-e836f31d9cd7_isr_946180_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/21c01109-d822-4f8f-9c15-e836f31d9cd7_isr_946180_32.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x15230b094590 state=pending>
parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x15230bf97290 state=pending>
parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x15230bf968d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 3765022f60a11db9bd350136cb5768e9
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e36c594a-b9bc-4cd7-a3c1-59e424792a69_isr_946072_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e36c594a-b9bc-4cd7-a3c1-59e424792a69_isr_946072_45.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x15230bf96a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x15230ac7d310 state=pending>
parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x15230ac7cc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 4a048eb96d666190d725dd2543841b9d
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e5819ef-e110-441a-8a5c-b4a23756d079_isr_946073_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e5819ef-e110-441a-8a5c-b4a23756d079_isr_946073_10.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x152310ea0e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x15230bc29e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x15230bc28a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 248e68c22c23e4e5b3ee17ef07065803
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f2a3047e-a0ed-4bd3-8694-297ff2c6428b_isr_946180_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f2a3047e-a0ed-4bd3-8694-297ff2c6428b_isr_946180_16.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x15230bc29490 state=pending>
parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x15230bc28f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x15230ab8af90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 5eb72e8574a68077db490fad06eaf4ac
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6206ed07-77cf-46de-a47f-6441a3ca4a5c_isr_946072_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6206ed07-77cf-46de-a47f-6441a3ca4a5c_isr_946072_10.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x15230ab8b410 state=pending>
parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x15231b2d7d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x15231b2d75d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 623ae8612047447831b7cf9572f6808e
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/21dda898-c035-4543-abdb-339d0caa6750_isr_946073_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/21dda898-c035-4543-abdb-339d0caa6750_isr_946073_56.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x15231b2d79d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x15231b2d6b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x15231b2d5b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 1a3d96fc46b1e5a7d8504fbc0a84e4e2
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fccd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a5fd5f33-1348-4678-8c46-7d339360ed90_isr_946182_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a5fd5f33-1348-4678-8c46-7d339360ed90_isr_946182_39.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x15231b2d57d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x15231b2d4690 state=pending>
parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x15231b2d6dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 18ef1915fccc9534d017d0b429dab6ae
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bfe9b614-8b2b-4c3c-aa51-4dc3e412df6a_isr_946072_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bfe9b614-8b2b-4c3c-aa51-4dc3e412df6a_isr_946072_55.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x15231b2d4450 state=pending>
parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x15231b266f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x15231b2667d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash f202a016376af973e55cc8a5ff655447
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ec384bf3-a8d8-475b-957e-05ffe0b19e83_isr_946180_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ec384bf3-a8d8-475b-957e-05ffe0b19e83_isr_946180_62.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x15231b266450 state=pending>
parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x15231b265490 state=pending>
parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: <AppFuture at 0x15231b264e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 186 has memoization hash f192ac424acacdcb302246ad476b0aca
parsl.dataflow.memoization INFO: Task 186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4f964645-ba71-4069-ae5c-75ca4a71de3c_isr_946073_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4f964645-ba71-4069-ae5c-75ca4a71de3c_isr_946073_21.stderr
parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186
parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: <AppFuture at 0x15231b265e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187
parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: <AppFuture at 0x15230be263d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: <AppFuture at 0x15230be26650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 2823ee5692121c01f36612147b5d9e1f
parsl.dataflow.memoization INFO: Task 189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bcbd1ee5-8f5b-495a-be47-58a11cc2269b_isr_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bcbd1ee5-8f5b-495a-be47-58a11cc2269b_isr_946181_31.stderr
parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189
parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: <AppFuture at 0x15230be24210 state=pending>
parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190
parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: <AppFuture at 0x15230afec450 state=pending>
parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: <AppFuture at 0x15230afec150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 192 has memoization hash bd8ded10dd489bbe3ac30afcedac21b7
parsl.dataflow.memoization INFO: Task 192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8a51dbc0-59b3-42dc-9d62-255fbddb829a_isr_946182_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8a51dbc0-59b3-42dc-9d62-255fbddb829a_isr_946182_23.stderr
parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192
parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: <AppFuture at 0x15230be261d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193
parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: <AppFuture at 0x15230b43b450 state=pending>
parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: <AppFuture at 0x15230b438810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 195 has memoization hash bcf4782a0ed40da29b938f4aeb0f0999
parsl.dataflow.memoization INFO: Task 195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/7addc496-aaba-4ba8-8d1c-15feabbe1443_isr_946073_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/7addc496-aaba-4ba8-8d1c-15feabbe1443_isr_946073_14.stderr
parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195
parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: <AppFuture at 0x15230b43b090 state=pending>
parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196
parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: <AppFuture at 0x15230bc60250 state=pending>
parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: <AppFuture at 0x15230bc60fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 20628bec6ac46f43ec6eb86f9530e5ba
parsl.dataflow.memoization INFO: Task 198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82a07f45-93fb-40fe-ba9e-4b77c3fb3197_isr_946181_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82a07f45-93fb-40fe-ba9e-4b77c3fb3197_isr_946181_5.stderr
parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198
parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: <AppFuture at 0x15230bc620d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199
parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: <AppFuture at 0x15230a214fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: <AppFuture at 0x15231b2cb750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 96612c0e248e2a4bb05a2c5b59e173bf
parsl.dataflow.memoization INFO: Task 201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b165e9ba-0238-44c5-a051-a7bb0e28ab60_isr_946073_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b165e9ba-0238-44c5-a051-a7bb0e28ab60_isr_946073_50.stderr
parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201
parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: <AppFuture at 0x15231b2cb5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202
parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: <AppFuture at 0x15231b2cad10 state=pending>
parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: <AppFuture at 0x15231b2c9e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 204 has memoization hash ba5b26490ba1c3925a897a3e880134cd
parsl.dataflow.memoization INFO: Task 204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4204b4ce-6686-4a6c-b819-d8f9c88bb648_isr_946180_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4204b4ce-6686-4a6c-b819-d8f9c88bb648_isr_946180_59.stderr
parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204
parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: <AppFuture at 0x15231b2c8c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205
parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: <AppFuture at 0x15231b2c9090 state=pending>
parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: <AppFuture at 0x15231b2c83d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 207 has memoization hash ff38d667c190e5533e751ee60b0fca31
parsl.dataflow.memoization INFO: Task 207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fceca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1170d194-9bad-47c1-a16d-585d31b40bae_isr_946182_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1170d194-9bad-47c1-a16d-585d31b40bae_isr_946182_36.stderr
parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207
parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: <AppFuture at 0x15231b2c9410 state=pending>
parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208
parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: <AppFuture at 0x15230bb0c5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: <AppFuture at 0x15230bb0c350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 43f048d253ac2504f3c05ea54c89eff7
parsl.dataflow.memoization INFO: Task 210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fbff1017-9f6e-4005-a5d5-2d81abb0a953_isr_946072_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fbff1017-9f6e-4005-a5d5-2d81abb0a953_isr_946072_15.stderr
parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210
parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: <AppFuture at 0x15230bcd3710 state=pending>
parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211
parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: <AppFuture at 0x15230bcd1e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: <AppFuture at 0x15230b7b9050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 213 has memoization hash e5054b9fc301b7afbb1bb2013af2a5de
parsl.dataflow.memoization INFO: Task 213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e347a3a3-0895-4ef3-b5ad-db3337a2e1eb_isr_946180_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e347a3a3-0895-4ef3-b5ad-db3337a2e1eb_isr_946180_9.stderr
parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213
parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: <AppFuture at 0x15230b7b88d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214
parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: <AppFuture at 0x15230b7b94d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: <AppFuture at 0x15230c741710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 216 has memoization hash dc9002303cb9d8ab5ff636355d750caa
parsl.dataflow.memoization INFO: Task 216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/543105da-58b2-40de-8150-de2b9f66a2c1_isr_946180_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/543105da-58b2-40de-8150-de2b9f66a2c1_isr_946180_39.stderr
parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216
parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: <AppFuture at 0x15230c742bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217
parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: <AppFuture at 0x15230c740610 state=pending>
parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: <AppFuture at 0x15230c7408d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 219 has memoization hash ab73ef9dcc19cfb8d600483f05ef3148
parsl.dataflow.memoization INFO: Task 219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c06ed5be-a5e4-4db4-95de-2b1a0c50cbfb_isr_946181_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c06ed5be-a5e4-4db4-95de-2b1a0c50cbfb_isr_946181_35.stderr
parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219
parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: <AppFuture at 0x15230c742e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220
parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: <AppFuture at 0x15230c742fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: <AppFuture at 0x15230bf0cfd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 1f12bbe5a7dfb5aecbb8cc475e03a4a9
parsl.dataflow.memoization INFO: Task 222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2a78194a-ab57-4b52-a55e-ddbebcb01f39_isr_946181_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2a78194a-ab57-4b52-a55e-ddbebcb01f39_isr_946181_28.stderr
parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222
parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: <AppFuture at 0x15230bf0e590 state=pending>
parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223
parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: <AppFuture at 0x15230bf0f410 state=pending>
parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: <AppFuture at 0x15230bf0dd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 5f5bc58ed70a8df58678830d0822ac85
parsl.dataflow.memoization INFO: Task 225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/aaa38cd5-74f0-43fe-8f49-9c4d118541bb_isr_946074_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/aaa38cd5-74f0-43fe-8f49-9c4d118541bb_isr_946074_56.stderr
parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225
parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: <AppFuture at 0x15230bf0d290 state=pending>
parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226
parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: <AppFuture at 0x15230bf0ce50 state=pending>
parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: <AppFuture at 0x15230bf0c110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 228 has memoization hash b4e71528c4183533b0110a8c587948a8
parsl.dataflow.memoization INFO: Task 228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d69c93f5-dd03-4afd-9af7-fce3330509be_isr_946181_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d69c93f5-dd03-4afd-9af7-fce3330509be_isr_946181_32.stderr
parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228
parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: <AppFuture at 0x15230bf0e210 state=pending>
parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229
parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: <AppFuture at 0x15231b2bb810 state=pending>
parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: <AppFuture at 0x15231b2bac10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 231 has memoization hash c840559757f9ffd0fc2a896cb47295d0
parsl.dataflow.memoization INFO: Task 231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a1cf5248-bb8d-4759-9829-b433ac987ef3_isr_946182_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a1cf5248-bb8d-4759-9829-b433ac987ef3_isr_946182_22.stderr
parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231
parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: <AppFuture at 0x15231b2baf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232
parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: <AppFuture at 0x15231b2b9950 state=pending>
parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: <AppFuture at 0x15231b2b9210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 462d514b9d7dc63947c0109525975952
parsl.dataflow.memoization INFO: Task 234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3dfd289-babb-4642-92b4-92d99f5f17a7_isr_946072_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3dfd289-babb-4642-92b4-92d99f5f17a7_isr_946072_3.stderr
parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234
parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: <AppFuture at 0x15231b2b82d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235
parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: <AppFuture at 0x15230c6d8610 state=pending>
parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: <AppFuture at 0x15230b4458d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 24fe9c4215def8d5afb5c00663c4f856
parsl.dataflow.memoization INFO: Task 237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c4b61661-4e98-4f84-b0dd-57ce27a1cd99_isr_946181_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c4b61661-4e98-4f84-b0dd-57ce27a1cd99_isr_946181_25.stderr
parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237
parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: <AppFuture at 0x15230b447110 state=pending>
parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238
parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: <AppFuture at 0x15230a38f190 state=pending>
parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: <AppFuture at 0x15230a38cc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 240 has memoization hash fb03623f3500a0d1b5541797f5dd8e85
parsl.dataflow.memoization INFO: Task 240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/81c3c77b-1ca0-491a-a427-00e95949a107_isr_946182_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/81c3c77b-1ca0-491a-a427-00e95949a107_isr_946182_11.stderr
parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240
parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: <AppFuture at 0x15230b445310 state=pending>
parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241
parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: <AppFuture at 0x15230ae80250 state=pending>
parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: <AppFuture at 0x15231b33fb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 3bde39877181e10664dcc1339a290554
parsl.dataflow.memoization INFO: Task 243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/436948bb-cbdf-4b46-befb-411f4817efbe_isr_946072_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/436948bb-cbdf-4b46-befb-411f4817efbe_isr_946072_14.stderr
parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243
parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: <AppFuture at 0x15231b33f990 state=pending>
parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244
parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: <AppFuture at 0x15231b33f0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: <AppFuture at 0x15231b33e0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 0127e05fb1836e9e35b50d08e4ac828b
parsl.dataflow.memoization INFO: Task 246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/deb10df4-76e0-4255-a340-2e28e519cf20_isr_946072_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/deb10df4-76e0-4255-a340-2e28e519cf20_isr_946072_28.stderr
parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246
parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: <AppFuture at 0x15231b33e450 state=pending>
parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247
parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: <AppFuture at 0x15231b33d550 state=pending>
parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: <AppFuture at 0x15231b33c4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 249 has memoization hash bb937feed5176a5f97ac350e183046af
parsl.dataflow.memoization INFO: Task 249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/19a32372-bf00-4ca4-a27a-54498b3fe4dc_isr_946073_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/19a32372-bf00-4ca4-a27a-54498b3fe4dc_isr_946073_16.stderr
parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249
parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: <AppFuture at 0x15231b33c1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250
parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: <AppFuture at 0x15230bf17050 state=pending>
parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: <AppFuture at 0x15230bf16e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 252 has memoization hash ac199d6f872ea36b507fe766ae107039
parsl.dataflow.memoization INFO: Task 252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/85d53ed5-7a47-4b23-8bff-8984d4fb20d2_isr_946180_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/85d53ed5-7a47-4b23-8bff-8984d4fb20d2_isr_946180_33.stderr
parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252
parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: <AppFuture at 0x15230bf172d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253
parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: <AppFuture at 0x15230bf15a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: <AppFuture at 0x15230bf17e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 255 has memoization hash c5df2557b78f285bfd6849eb1282065b
parsl.dataflow.memoization INFO: Task 255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ee920608-b070-4cee-adc3-3be63698ca62_isr_946180_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ee920608-b070-4cee-adc3-3be63698ca62_isr_946180_51.stderr
parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255
parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: <AppFuture at 0x15230bf15790 state=pending>
parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256
parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: <AppFuture at 0x15231b2b7dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: <AppFuture at 0x15231b2b7290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 258 has memoization hash bbbaa58988e6ea28ccd7793708f5ec08
parsl.dataflow.memoization INFO: Task 258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8c806c12-92c2-40ee-a082-775969ac827e_isr_946182_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8c806c12-92c2-40ee-a082-775969ac827e_isr_946182_30.stderr
parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258
parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: <AppFuture at 0x15230a4f7f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259
parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: <AppFuture at 0x15231b2b6190 state=pending>
parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: <AppFuture at 0x15231b2b5810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 261 has memoization hash db3de542c3c00966432d3dbee30137b5
parsl.dataflow.memoization INFO: Task 261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d8dd6500-509d-41ce-9253-57491381d36f_isr_946074_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d8dd6500-509d-41ce-9253-57491381d36f_isr_946074_38.stderr
parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261
parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: <AppFuture at 0x15231b2b6850 state=pending>
parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262
parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: <AppFuture at 0x15231b2b4250 state=pending>
parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: <AppFuture at 0x15231b2b4090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 264 has memoization hash f881c3fd8f191e2c42fdd151ffd5f1f8
parsl.dataflow.memoization INFO: Task 264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/84d3eabc-b73e-47ef-99d7-d0b74e2f927c_isr_946074_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/84d3eabc-b73e-47ef-99d7-d0b74e2f927c_isr_946074_6.stderr
parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264
parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: <AppFuture at 0x152310874b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265
parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: <AppFuture at 0x152310877d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: <AppFuture at 0x152310877290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 8b58634714295803df128c429dd39172
parsl.dataflow.memoization INFO: Task 267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8e36f6a3-0659-4fc5-b2b7-b3caafc7fbc8_isr_946073_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8e36f6a3-0659-4fc5-b2b7-b3caafc7fbc8_isr_946073_5.stderr
parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267
parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: <AppFuture at 0x152310876e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268
parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: <AppFuture at 0x15230a60fad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: <AppFuture at 0x15230a378b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 270 has memoization hash d74af8dc33f61788ecfb7d012c6740ed
parsl.dataflow.memoization INFO: Task 270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e666065e-992e-40eb-99cd-238496412a85_isr_946180_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e666065e-992e-40eb-99cd-238496412a85_isr_946180_5.stderr
parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270
parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: <AppFuture at 0x152310874e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271
parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: <AppFuture at 0x15231b273d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: <AppFuture at 0x15231b272c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 449e1b089351e8a67dbbc56d2a2e411d
parsl.dataflow.memoization INFO: Task 273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9d0cbb84-14ef-44a3-81cb-a8ef1d21ecd6_isr_946073_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9d0cbb84-14ef-44a3-81cb-a8ef1d21ecd6_isr_946073_1.stderr
parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273
parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: <AppFuture at 0x15231b272690 state=pending>
parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274
parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: <AppFuture at 0x15231b271b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: <AppFuture at 0x15231b271210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 5de9e88ccc7a7c28ea1d0f31c45fc9ec
parsl.dataflow.memoization INFO: Task 276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b2aee601-7495-4d94-a497-a5241957fd51_isr_946073_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b2aee601-7495-4d94-a497-a5241957fd51_isr_946073_49.stderr
parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276
parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: <AppFuture at 0x15231b270690 state=pending>
parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277
parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: <AppFuture at 0x15231b270a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: <AppFuture at 0x15230a7bd950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 279 has memoization hash f9e5e75265fc0bfe8607a1f5e79f538e
parsl.dataflow.memoization INFO: Task 279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fa95802-4f58-46b9-96af-3ae47ea2fc33_isr_946072_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fa95802-4f58-46b9-96af-3ae47ea2fc33_isr_946072_16.stderr
parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279
parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: <AppFuture at 0x15230be24110 state=pending>
parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280
parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: <AppFuture at 0x15230bc07ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: <AppFuture at 0x15230bc076d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 2769e650d5a982f6153d37ac39e72be9
parsl.dataflow.memoization INFO: Task 282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c1a92695-b2b4-481b-becd-e05e2d30b91b_isr_946072_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c1a92695-b2b4-481b-becd-e05e2d30b91b_isr_946072_8.stderr
parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282
parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: <AppFuture at 0x15230bc05a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283
parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: <AppFuture at 0x15230b5e6cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: <AppFuture at 0x15230b5e4250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 285 has memoization hash cfe4ddc5faaae9d9851c135f90f4b645
parsl.dataflow.memoization INFO: Task 285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/de6b5108-90a4-4748-8c95-7bc62453f7af_isr_946072_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/de6b5108-90a4-4748-8c95-7bc62453f7af_isr_946072_29.stderr
parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285
parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: <AppFuture at 0x15230b5e7f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286
parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: <AppFuture at 0x15231b336cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: <AppFuture at 0x15231b3367d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 288 has memoization hash ba65c9f5f12b099fd62b3674f28bdbd5
parsl.dataflow.memoization INFO: Task 288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a7d03e77-670a-4887-85a5-31d2f9da95cc_isr_946180_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a7d03e77-670a-4887-85a5-31d2f9da95cc_isr_946180_44.stderr
parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288
parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: <AppFuture at 0x15231b3360d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289
parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: <AppFuture at 0x15231b335790 state=pending>
parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: <AppFuture at 0x15231b334e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 291 has memoization hash f3d2ab122ec5ab2ffbbe286e7653ae76
parsl.dataflow.memoization INFO: Task 291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b4755169-fedd-4eb5-b736-b13522655465_isr_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b4755169-fedd-4eb5-b736-b13522655465_isr_946073_61.stderr
parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291
parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: <AppFuture at 0x15231b335e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292
parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: <AppFuture at 0x15231b33c050 state=pending>
parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: <AppFuture at 0x15230a935110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 294 has memoization hash cdc64e0bd5fdb0003edc8cea0a1aaaa9
parsl.dataflow.memoization INFO: Task 294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/39337a3c-c2e1-4eba-9980-a643e34a245e_isr_946072_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/39337a3c-c2e1-4eba-9980-a643e34a245e_isr_946072_41.stderr
parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294
parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: <AppFuture at 0x15231b28bd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295
parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: <AppFuture at 0x15231b28ad50 state=pending>
parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: <AppFuture at 0x15231b28a750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 7a86381f43fd0be69e8196fd3e9d7fe6
parsl.dataflow.memoization INFO: Task 297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/bb97410f-f6f1-4a5c-9b0c-7691a16d2f35_isr_946180_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/bb97410f-f6f1-4a5c-9b0c-7691a16d2f35_isr_946180_52.stderr
parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297
parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: <AppFuture at 0x15231b28aad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298
parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: <AppFuture at 0x15231b289150 state=pending>
parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: <AppFuture at 0x15231b288c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 567e92f5b30de4575ba9798d130424d5
parsl.dataflow.memoization INFO: Task 300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2e0c1bb3-f968-4ab4-8f11-4bfd131a8c9b_isr_946182_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2e0c1bb3-f968-4ab4-8f11-4bfd131a8c9b_isr_946182_59.stderr
parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300
parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: <AppFuture at 0x15231b2887d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301
parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: <AppFuture at 0x15230c8d1050 state=pending>
parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: <AppFuture at 0x15230c8d1950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 1ab9b00e6b8d690a701792ee57277e90
parsl.dataflow.memoization INFO: Task 303 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/df29ff19-ac0d-471d-a6e3-f57189501125_isr_946182_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/df29ff19-ac0d-471d-a6e3-f57189501125_isr_946182_4.stderr
parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303
parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: <AppFuture at 0x15230c8d3290 state=pending>
parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304
parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: <AppFuture at 0x15230c8d3410 state=pending>
parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: <AppFuture at 0x15230c8d0810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 79b31e1c0036571bc7c769711b13fecc
parsl.dataflow.memoization INFO: Task 306 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ae576817-f9f0-4f64-a53b-0302f9afb70e_isr_946074_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ae576817-f9f0-4f64-a53b-0302f9afb70e_isr_946074_53.stderr
parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306
parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: <AppFuture at 0x15230c8d2a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307
parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: <AppFuture at 0x15230c8d1e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: <AppFuture at 0x15230c0d1190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 2b85e93ccead4e34346306d6faa4cb15
parsl.dataflow.memoization INFO: Task 309 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/69a737c6-c9f1-45dc-a201-78c32dbbab6c_isr_946181_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/69a737c6-c9f1-45dc-a201-78c32dbbab6c_isr_946181_3.stderr
parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309
parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: <AppFuture at 0x15230c0d3090 state=pending>
parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310
parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: <AppFuture at 0x15230c0d2c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: <AppFuture at 0x15230c0d3510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 9a921c85cd12ea5e006e851f29c48c35
parsl.dataflow.memoization INFO: Task 312 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ee2b8646-f5cc-450c-be1c-7a56887999d0_isr_946074_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ee2b8646-f5cc-450c-be1c-7a56887999d0_isr_946074_11.stderr
parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312
parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: <AppFuture at 0x15230c0d3950 state=pending>
parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313
parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: <AppFuture at 0x15230a31ad90 state=pending>
parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: <AppFuture at 0x15230a31afd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 2559ee7238175cda75041b819d592d2b
parsl.dataflow.memoization INFO: Task 315 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c825bfa7-d14a-4e30-868e-34bed79118bd_isr_946180_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c825bfa7-d14a-4e30-868e-34bed79118bd_isr_946180_46.stderr
parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315
parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: <AppFuture at 0x15230a31a750 state=pending>
parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316
parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: <AppFuture at 0x15230c6e9390 state=pending>
parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: <AppFuture at 0x15230c6ea350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 8b707eaa4a4c6dfced2d4242dfa945cc
parsl.dataflow.memoization INFO: Task 318 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f76d25c4-19b5-44b1-a272-57a86585b4af_isr_946182_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f76d25c4-19b5-44b1-a272-57a86585b4af_isr_946182_62.stderr
parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318
parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: <AppFuture at 0x15230c6ebcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319
parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: <AppFuture at 0x15230c6ead50 state=pending>
parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: <AppFuture at 0x15230c6e97d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 375c4f527f0c6ef9bfa30272e95757a3
parsl.dataflow.memoization INFO: Task 321 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b7ef4c01-3385-4ed2-a0e5-89441678f3b7_isr_946180_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b7ef4c01-3385-4ed2-a0e5-89441678f3b7_isr_946180_3.stderr
parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321
parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: <AppFuture at 0x15230c6ebe10 state=pending>
parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322
parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: <AppFuture at 0x15230c6eb390 state=pending>
parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: <AppFuture at 0x15231b32be50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 800a3e3c0be7d0d59e4292779c71f0ad
parsl.dataflow.memoization INFO: Task 324 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/34d0c732-cce5-4c71-ba30-04c3ad034ede_isr_946180_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/34d0c732-cce5-4c71-ba30-04c3ad034ede_isr_946180_29.stderr
parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324
parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: <AppFuture at 0x15230c6eb590 state=pending>
parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325
parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: <AppFuture at 0x15231b32a890 state=pending>
parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: <AppFuture at 0x15231b32a390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 216a24e2f684a0c39e7bb941792abdcf
parsl.dataflow.memoization INFO: Task 327 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/81ad036b-5580-415d-9ab2-fa3bbf01d770_isr_946073_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/81ad036b-5580-415d-9ab2-fa3bbf01d770_isr_946073_15.stderr
parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327
parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: <AppFuture at 0x15231b32a710 state=pending>
parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328
parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: <AppFuture at 0x15231b329690 state=pending>
parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: <AppFuture at 0x15231b328a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 7de70fcb220d2a588b0a924e744cbbe9
parsl.dataflow.memoization INFO: Task 330 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ad19903d-f067-4175-a652-b94755ae9eb1_isr_946072_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ad19903d-f067-4175-a652-b94755ae9eb1_isr_946072_51.stderr
parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330
parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: <AppFuture at 0x15231b328c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331
parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: <AppFuture at 0x15230ba17290 state=pending>
parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: <AppFuture at 0x15231b248f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 300a8f9f2549d782243d5bf484d6d53b
parsl.dataflow.memoization INFO: Task 333 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/198ae5a7-70de-4e22-9aac-7c9d9cade973_isr_946180_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/198ae5a7-70de-4e22-9aac-7c9d9cade973_isr_946180_31.stderr
parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333
parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: <AppFuture at 0x15231b32ac90 state=pending>
parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334
parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: <AppFuture at 0x15231b248450 state=pending>
parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: <AppFuture at 0x15231b24b710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 336 has memoization hash e309069bbf92c3d1f56ecc03b2418ca4
parsl.dataflow.memoization INFO: Task 336 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/eca15dec-3612-40af-a634-381af6f394de_isr_946181_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/eca15dec-3612-40af-a634-381af6f394de_isr_946181_13.stderr
parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336
parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: <AppFuture at 0x15231b24a950 state=pending>
parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337
parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: <AppFuture at 0x15231b24a790 state=pending>
parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: <AppFuture at 0x15231b24a210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 339 has memoization hash a2179b4b6af66341d6458da472718381
parsl.dataflow.memoization INFO: Task 339 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7ee4f318-f71e-4303-b2a2-bd7df1fcca5c_isr_946180_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7ee4f318-f71e-4303-b2a2-bd7df1fcca5c_isr_946180_55.stderr
parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339
parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: <AppFuture at 0x15231b249ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340
parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: <AppFuture at 0x15231b249390 state=pending>
parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: <AppFuture at 0x15230a4e6350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 30c242eb738917c56ebccf1ea3f16c02
parsl.dataflow.memoization INFO: Task 342 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/dd09dc3c-63dc-4365-9dfc-874ffd4e473e_isr_946181_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/dd09dc3c-63dc-4365-9dfc-874ffd4e473e_isr_946181_21.stderr
parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342
parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: <AppFuture at 0x15231b249110 state=pending>
parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343
parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: <AppFuture at 0x15230bfbf190 state=pending>
parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: <AppFuture at 0x15230bfbf390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 5cdadc5d8dc949d5b7855cda750cea0d
parsl.dataflow.memoization INFO: Task 345 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/dcccff5a-9f8a-4a95-9f6c-efd1195155c6_isr_946074_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/dcccff5a-9f8a-4a95-9f6c-efd1195155c6_isr_946074_50.stderr
parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345
parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: <AppFuture at 0x15230bfbc110 state=pending>
parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346
parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: <AppFuture at 0x15230bfbef10 state=pending>
parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: <AppFuture at 0x15230bfbc790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 348 has memoization hash feafefa3e84205cc7978c9b9f2b2ba1a
parsl.dataflow.memoization INFO: Task 348 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/090d5f59-40ea-4385-8aab-20a883699e84_isr_946180_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/090d5f59-40ea-4385-8aab-20a883699e84_isr_946180_56.stderr
parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348
parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: <AppFuture at 0x15230bfbe990 state=pending>
parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349
parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: <AppFuture at 0x15230b369990 state=pending>
parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: <AppFuture at 0x15230b369310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 29080d88610ef2a59102572c77b37b33
parsl.dataflow.memoization INFO: Task 351 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/87c631f4-6d58-4662-a12a-67ef1e43df89_isr_946072_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/87c631f4-6d58-4662-a12a-67ef1e43df89_isr_946072_26.stderr
parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351
parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: <AppFuture at 0x15230b368950 state=pending>
parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352
parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: <AppFuture at 0x15230bfe18d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: <AppFuture at 0x15230bfe3b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 354 has memoization hash f2190e68af0ff6b51885f9689c2b5139
parsl.dataflow.memoization INFO: Task 354 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d3f36c0e-1a16-44ca-8ad9-492106cf1c37_isr_946074_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d3f36c0e-1a16-44ca-8ad9-492106cf1c37_isr_946074_44.stderr
parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354
parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: <AppFuture at 0x15230bfe3f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355
parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: <AppFuture at 0x15230bfe0290 state=pending>
parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: <AppFuture at 0x15230bfe0910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 8c16d54db2431f2e0016cf2d969a49de
parsl.dataflow.memoization INFO: Task 357 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/b32e454b-32ce-4759-a3fb-75be3b5077c6_isr_946072_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/b32e454b-32ce-4759-a3fb-75be3b5077c6_isr_946072_38.stderr
parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357
parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: <AppFuture at 0x15230bfe1010 state=pending>
parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358
parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: <AppFuture at 0x15230c7450d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: <AppFuture at 0x15230c747650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 360 has memoization hash ec8f95d64f3540d7bf87b10a64b043bc
parsl.dataflow.memoization INFO: Task 360 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/932424ee-12e7-443a-9e25-d546e910c0d0_isr_946180_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/932424ee-12e7-443a-9e25-d546e910c0d0_isr_946180_12.stderr
parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360
parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: <AppFuture at 0x15230c745610 state=pending>
parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361
parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: <AppFuture at 0x15230c745590 state=pending>
parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: <AppFuture at 0x15230c746fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 595da09c18c7c34cc8fc8e2f955645ad
parsl.dataflow.memoization INFO: Task 363 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9eae70a2-1baf-4d7b-b3b5-2d6236f776c5_isr_946180_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9eae70a2-1baf-4d7b-b3b5-2d6236f776c5_isr_946180_37.stderr
parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363
parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: <AppFuture at 0x15230c746a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364
parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: <AppFuture at 0x152311077450 state=pending>
parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: <AppFuture at 0x152311075710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 5440c270e641eb9603efe786e5e7f92a
parsl.dataflow.memoization INFO: Task 366 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/cbda7158-21b5-45a2-ba39-1437971d5587_isr_946074_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/cbda7158-21b5-45a2-ba39-1437971d5587_isr_946074_9.stderr
parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366
parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: <AppFuture at 0x1523110754d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367
parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: <AppFuture at 0x152311076d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: <AppFuture at 0x152311074e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 369 has memoization hash ca9d1e8c5db60a22b78b4ff334de672a
parsl.dataflow.memoization INFO: Task 369 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a45a54fe-a897-4fc5-b0de-de48457fac79_isr_946074_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a45a54fe-a897-4fc5-b0de-de48457fac79_isr_946074_26.stderr
parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369
parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: <AppFuture at 0x1523110756d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370
parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: <AppFuture at 0x1523110772d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: <AppFuture at 0x15230bfcd8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 66948aa662ad0db74f32dbd4422d3cdf
parsl.dataflow.memoization INFO: Task 372 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb981c36-e331-48c1-8084-ad6bae302491_isr_946182_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb981c36-e331-48c1-8084-ad6bae302491_isr_946182_24.stderr
parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372
parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: <AppFuture at 0x15230bfcdad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373
parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: <AppFuture at 0x15230bfcd750 state=pending>
parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: <AppFuture at 0x15230bfcd810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 375 has memoization hash ea3b7eb080269d7c403999bb48f3d3e2
parsl.dataflow.memoization INFO: Task 375 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b2ec709-fbd9-4e1b-b38c-5f7b0f9a71fe_isr_946180_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b2ec709-fbd9-4e1b-b38c-5f7b0f9a71fe_isr_946180_8.stderr
parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375
parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: <AppFuture at 0x15230bfcd2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376
parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: <AppFuture at 0x15230bcdbf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: <AppFuture at 0x15231b283c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 1647105b345b17141d3e1eb1d3e73ea9
parsl.dataflow.memoization INFO: Task 378 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b8e30b77-ca47-445f-8875-e3a4798dafc0_isr_946073_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b8e30b77-ca47-445f-8875-e3a4798dafc0_isr_946073_45.stderr
parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378
parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: <AppFuture at 0x15231b283f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379
parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: <AppFuture at 0x15231b2830d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: <AppFuture at 0x15231b282190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 7c5a1c4f2864e727afdd55648f127e5f
parsl.dataflow.memoization INFO: Task 381 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/28ffa85f-f5c0-45ba-90bb-c33a24a0050d_isr_946073_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/28ffa85f-f5c0-45ba-90bb-c33a24a0050d_isr_946073_36.stderr
parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381
parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: <AppFuture at 0x15231b282750 state=pending>
parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382
parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: <AppFuture at 0x15231b281c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: <AppFuture at 0x15231b280910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 2095af55dc12a4c85fc2c0734989e448
parsl.dataflow.memoization INFO: Task 384 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6e447186-8036-43f1-8ba9-e73ce5fdb348_isr_946073_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6e447186-8036-43f1-8ba9-e73ce5fdb348_isr_946073_60.stderr
parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384
parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: <AppFuture at 0x15231b280390 state=pending>
parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385
parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: <AppFuture at 0x15231b282b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: <AppFuture at 0x15230b847490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 09d4e1e7e5aad0af5c541ed1b0da0577
parsl.dataflow.memoization INFO: Task 387 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/42d694db-a714-44b2-98f4-021411463551_isr_946072_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/42d694db-a714-44b2-98f4-021411463551_isr_946072_57.stderr
parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387
parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: <AppFuture at 0x15230b847d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388
parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: <AppFuture at 0x15230b5360d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: <AppFuture at 0x15230b5342d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 09d1b2e54106462971bc7a24ad81a7ee
parsl.dataflow.memoization INFO: Task 390 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d3f4f9bd-64ff-4bd9-b41d-f5b322aafbf9_isr_946072_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d3f4f9bd-64ff-4bd9-b41d-f5b322aafbf9_isr_946072_21.stderr
parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390
parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: <AppFuture at 0x15231b281c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391
parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: <AppFuture at 0x15230b0fa410 state=pending>
parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: <AppFuture at 0x15230b0f8490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 393 has memoization hash bfbe64c69bf198d85c2c1bdb951d3db5
parsl.dataflow.memoization INFO: Task 393 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/30296992-c294-4dd6-a2ef-eed532683c3e_isr_946181_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/30296992-c294-4dd6-a2ef-eed532683c3e_isr_946181_37.stderr
parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393
parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: <AppFuture at 0x15230bc28950 state=pending>
parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394
parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: <AppFuture at 0x15230bec84d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: <AppFuture at 0x15230becb0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 598e3cc167dd0096b5206389257c8017
parsl.dataflow.memoization INFO: Task 396 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/be8c82c7-401a-4127-bde5-2f9ad37be993_isr_946181_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/be8c82c7-401a-4127-bde5-2f9ad37be993_isr_946181_51.stderr
parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396
parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: <AppFuture at 0x15230b0f8dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397
parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: <AppFuture at 0x15230af4bb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: <AppFuture at 0x15231ab6cfd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 399 has memoization hash ac7a5160fa775cff90b9ce7450394215
parsl.dataflow.memoization INFO: Task 399 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/11066b3a-e737-4ab3-b814-b2d21f68d560_isr_946181_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/11066b3a-e737-4ab3-b814-b2d21f68d560_isr_946181_50.stderr
parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399
parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: <AppFuture at 0x15230c0990d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400
parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: <AppFuture at 0x15231ab6c890 state=pending>
parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: <AppFuture at 0x15230b541290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 2ee6ff9e24a84e41eb508c987486ec2b
parsl.dataflow.memoization INFO: Task 402 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8acae5a7-8ca7-4f68-ae0f-08e7a4ebdf9e_isr_946074_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8acae5a7-8ca7-4f68-ae0f-08e7a4ebdf9e_isr_946074_17.stderr
parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402
parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: <AppFuture at 0x15231ab6c790 state=pending>
parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403
parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: <AppFuture at 0x15230b5402d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: <AppFuture at 0x15230b542250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 405 has memoization hash a4221bfc40ae340e3512208436094b6b
parsl.dataflow.memoization INFO: Task 405 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6ef07143-7bec-4116-a19a-2a6e9db28a37_isr_946182_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6ef07143-7bec-4116-a19a-2a6e9db28a37_isr_946182_48.stderr
parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405
parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: <AppFuture at 0x15230b540410 state=pending>
parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406
parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: <AppFuture at 0x152311057750 state=pending>
parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: <AppFuture at 0x152311057610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 926416b4d2d056eb05b4a2ed5a946187
parsl.dataflow.memoization INFO: Task 408 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3c103a23-ae40-435d-a860-2bcc75f25010_isr_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3c103a23-ae40-435d-a860-2bcc75f25010_isr_946182_2.stderr
parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408
parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: <AppFuture at 0x152311056990 state=pending>
parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409
parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: <AppFuture at 0x15231058dcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: <AppFuture at 0x15230fa055d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 7654fafb8934e20e6a40b68f1b17728f
parsl.dataflow.memoization INFO: Task 411 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/67a799da-bef2-4a35-a889-8093ee9da949_isr_946072_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/67a799da-bef2-4a35-a889-8093ee9da949_isr_946072_47.stderr
parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411
parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: <AppFuture at 0x1523110fac10 state=pending>
parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412
parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: <AppFuture at 0x152311044090 state=pending>
parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: <AppFuture at 0x152311049810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 9e4c0b4a2241fda7c3cee3edf21ca6e0
parsl.dataflow.memoization INFO: Task 414 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ae9b6ebf-b1b1-4b29-883e-ad634e03a19f_isr_946180_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ae9b6ebf-b1b1-4b29-883e-ad634e03a19f_isr_946180_17.stderr
parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414
parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: <AppFuture at 0x1523110494d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415
parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: <AppFuture at 0x15231104b310 state=pending>
parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: <AppFuture at 0x15231b2638d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 193158e2e270ddcabf05c85642d21c25
parsl.dataflow.memoization INFO: Task 417 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/7c4a1f2c-4cb9-4ffd-8a21-a0c6a34b0b0f_isr_946072_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/7c4a1f2c-4cb9-4ffd-8a21-a0c6a34b0b0f_isr_946072_54.stderr
parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417
parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: <AppFuture at 0x15231b2625d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418
parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: <AppFuture at 0x15231b262cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: <AppFuture at 0x15231b261dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 420 has memoization hash b54aca5532ae3090eedc6a317153c488
parsl.dataflow.memoization INFO: Task 420 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9b8e3c22-13b1-46e1-83bf-85f72ed093eb_isr_946072_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9b8e3c22-13b1-46e1-83bf-85f72ed093eb_isr_946072_35.stderr
parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420
parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: <AppFuture at 0x15231b262350 state=pending>
parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421
parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: <AppFuture at 0x15231b261150 state=pending>
parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: <AppFuture at 0x15231b2606d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 60543a566b72966b438220ffc41a705f
parsl.dataflow.memoization INFO: Task 423 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58d977e2-6e16-4662-ae76-69d07cb687f9_isr_946182_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58d977e2-6e16-4662-ae76-69d07cb687f9_isr_946182_13.stderr
parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423
parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: <AppFuture at 0x15231b260290 state=pending>
parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424
parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: <AppFuture at 0x15231b277b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: <AppFuture at 0x15231b277310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 8bebe3b98e248abef78f244662639d2a
parsl.dataflow.memoization INFO: Task 426 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a411c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/170a23e3-2c1d-41af-9918-b197dea035aa_isr_946074_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/170a23e3-2c1d-41af-9918-b197dea035aa_isr_946074_45.stderr
parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426
parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: <AppFuture at 0x15231b277250 state=pending>
parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427
parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: <AppFuture at 0x15231b2767d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: <AppFuture at 0x15231b275ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 706ac19756041f7611f7ee65402c938e
parsl.dataflow.memoization INFO: Task 429 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/685f34a1-a470-4f84-8836-66f376a7eeee_isr_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/685f34a1-a470-4f84-8836-66f376a7eeee_isr_946181_2.stderr
parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429
parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: <AppFuture at 0x15231b275710 state=pending>
parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430
parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: <AppFuture at 0x15231b274690 state=pending>
parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: <AppFuture at 0x15231b274e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 51fdccecdec2da4acb5ad8123c1be5a9
parsl.dataflow.memoization INFO: Task 432 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/2da0e428-eadb-4c04-8671-9dedd0295304_isr_946073_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/2da0e428-eadb-4c04-8671-9dedd0295304_isr_946073_19.stderr
parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432
parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: <AppFuture at 0x15231b275190 state=pending>
parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433
parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: <AppFuture at 0x15231b5e7e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: <AppFuture at 0x15231b5e4d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 435 has memoization hash f3139f59b33cfb7e27e0533e34d529f2
parsl.dataflow.memoization INFO: Task 435 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/f02f53a9-5347-4463-b6a7-7eeb59cd67db_isr_946074_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/f02f53a9-5347-4463-b6a7-7eeb59cd67db_isr_946074_14.stderr
parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435
parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: <AppFuture at 0x15231b5e72d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436
parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: <AppFuture at 0x15231b5e4cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: <AppFuture at 0x15231b5e63d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 26b7947d4e01d804e885ffbff005e5ca
parsl.dataflow.memoization INFO: Task 438 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/de7dce0b-1ad5-4896-ac73-4e73a27efe4d_isr_946182_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/de7dce0b-1ad5-4896-ac73-4e73a27efe4d_isr_946182_17.stderr
parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438
parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: <AppFuture at 0x15231b5e6790 state=pending>
parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439
parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: <AppFuture at 0x15231b5f8e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: <AppFuture at 0x15231b5faf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 6924149b001fbc15e0dd4cba9aedf364
parsl.dataflow.memoization INFO: Task 441 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6b8c4428-d70a-4dd2-a838-eab61204ec8c_isr_946181_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6b8c4428-d70a-4dd2-a838-eab61204ec8c_isr_946181_60.stderr
parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441
parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: <AppFuture at 0x15231b5f9410 state=pending>
parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442
parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: <AppFuture at 0x15231b5f9790 state=pending>
parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: <AppFuture at 0x15231b5f8490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 444 has memoization hash d0e9922210a954c7cafde3eebc39764f
parsl.dataflow.memoization INFO: Task 444 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/908b04bb-78b8-4f4e-addc-760c057a78a9_isr_946180_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/908b04bb-78b8-4f4e-addc-760c057a78a9_isr_946180_53.stderr
parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444
parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: <AppFuture at 0x15231b5fbbd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445
parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: <AppFuture at 0x152310e894d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: <AppFuture at 0x152310e8b890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 955621d4fe91b9d31cd33c8803a8137f
parsl.dataflow.memoization INFO: Task 447 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a427a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3bb30c9-4811-44ac-a0de-184ca981fdaf_isr_946072_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3bb30c9-4811-44ac-a0de-184ca981fdaf_isr_946072_49.stderr
parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447
parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: <AppFuture at 0x1523105b5390 state=pending>
parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448
parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: <AppFuture at 0x152310e8b0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: <AppFuture at 0x152310867010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 89489ab826d62a10080831f6f02d6a7e
parsl.dataflow.memoization INFO: Task 450 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e4feb5a6-5366-472b-85d9-6032dc78e1b0_isr_946181_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e4feb5a6-5366-472b-85d9-6032dc78e1b0_isr_946181_53.stderr
parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450
parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: <AppFuture at 0x152310866f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451
parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: <AppFuture at 0x152310867a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: <AppFuture at 0x15230e7e7810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 80a103817df8643e49f4b3b9d830cc4b
parsl.dataflow.memoization INFO: Task 453 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9b494846-9744-4535-9ecc-09a85a5a7986_isr_946181_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9b494846-9744-4535-9ecc-09a85a5a7986_isr_946181_52.stderr
parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453
parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: <AppFuture at 0x152310867dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454
parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: <AppFuture at 0x15230a1eddd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: <AppFuture at 0x15230a1ed5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 456 has memoization hash c0a2a076f3281a65738854fa847ad9f4
parsl.dataflow.memoization INFO: Task 456 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/00547a90-29f3-4f02-b9e2-95ccfae2cb46_isr_946180_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/00547a90-29f3-4f02-b9e2-95ccfae2cb46_isr_946180_54.stderr
parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456
parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: <AppFuture at 0x15230a1ef710 state=pending>
parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457
parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: <AppFuture at 0x15230a03f710 state=pending>
parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: <AppFuture at 0x15230a03e090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 63b8e5e2aa1b8d79b2fe57eb67626bf3
parsl.dataflow.memoization INFO: Task 459 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/138cac86-975a-497a-a3aa-f8d77c673783_isr_946182_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/138cac86-975a-497a-a3aa-f8d77c673783_isr_946182_9.stderr
parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459
parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: <AppFuture at 0x15230a03cd10 state=pending>
parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460
parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: <AppFuture at 0x15230a03ec10 state=pending>
parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: <AppFuture at 0x15230a03d710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 462 has memoization hash ef35022fa38c2c6535ca92cb0be622b8
parsl.dataflow.memoization INFO: Task 462 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d08dc57b-ae49-46e1-91d5-9f903f0c0424_isr_946181_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d08dc57b-ae49-46e1-91d5-9f903f0c0424_isr_946181_57.stderr
parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462
parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: <AppFuture at 0x15230a03d850 state=pending>
parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463
parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: <AppFuture at 0x15230a03c050 state=pending>
parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: <AppFuture at 0x15230a287410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 52fe1fbca6279907a3375bbab087419e
parsl.dataflow.memoization INFO: Task 465 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/12daaafb-7208-4f82-9663-245a9a1e274d_isr_946072_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/12daaafb-7208-4f82-9663-245a9a1e274d_isr_946072_24.stderr
parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465
parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: <AppFuture at 0x15230a03d590 state=pending>
parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466
parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: <AppFuture at 0x152310e8e390 state=pending>
parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: <AppFuture at 0x152310e8fcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 17a5fe1c8cbe9e13d3e28a1e82cb11d5
parsl.dataflow.memoization INFO: Task 468 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0ab26328-4b79-4914-9a2d-8c7e087612cb_isr_946182_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0ab26328-4b79-4914-9a2d-8c7e087612cb_isr_946182_35.stderr
parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468
parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: <AppFuture at 0x152310e8f690 state=pending>
parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469
parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: <AppFuture at 0x15231b253ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: <AppFuture at 0x15231b253090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 6063d6f9dc2926f418d50f05b6828cf6
parsl.dataflow.memoization INFO: Task 471 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/5f644685-f5ee-410d-85e5-e02ef9534af0_isr_946073_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/5f644685-f5ee-410d-85e5-e02ef9534af0_isr_946073_62.stderr
parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471
parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: <AppFuture at 0x15231b24a9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472
parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: <AppFuture at 0x15231b252690 state=pending>
parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: <AppFuture at 0x15231b251990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 47716c856664f36685df30d7cfbcc316
parsl.dataflow.memoization INFO: Task 474 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4a46f180-5752-47c7-97a4-2883a7fea544_isr_946074_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4a46f180-5752-47c7-97a4-2883a7fea544_isr_946074_46.stderr
parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474
parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: <AppFuture at 0x15231b251d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475
parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: <AppFuture at 0x15231b250390 state=pending>
parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: <AppFuture at 0x15231b253b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 477 has memoization hash bef71a53466719da3ce395ba07557b16
parsl.dataflow.memoization INFO: Task 477 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/70de7056-1273-4908-94e8-b94d48e36b35_isr_946072_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/70de7056-1273-4908-94e8-b94d48e36b35_isr_946072_48.stderr
parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477
parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: <AppFuture at 0x15231b260d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478
parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: <AppFuture at 0x15231b2503d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: <AppFuture at 0x15230a3e0390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 59fe389db4f03b67156d0f0653452a7b
parsl.dataflow.memoization INFO: Task 480 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c01b65b3-7c1b-4569-906e-30e9443a201a_isr_946180_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c01b65b3-7c1b-4569-906e-30e9443a201a_isr_946180_20.stderr
parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480
parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: <AppFuture at 0x15231056c850 state=pending>
parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481
parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: <AppFuture at 0x15230a3e2e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: <AppFuture at 0x15230a01ac10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 07a8559d17f5a86fcdb1c3caad1ff3d0
parsl.dataflow.memoization INFO: Task 483 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/757b25ce-f0c5-40a9-9eca-d17cdc708e2d_isr_946073_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/757b25ce-f0c5-40a9-9eca-d17cdc708e2d_isr_946073_27.stderr
parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483
parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: <AppFuture at 0x15231074bf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484
parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: <AppFuture at 0x15230a018690 state=pending>
parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: <AppFuture at 0x15230a01b590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 7723327fecc40d9f80cee257dc9caeed
parsl.dataflow.memoization INFO: Task 486 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c54a0c1d-878d-4ead-828c-ad1f5f3335fd_isr_946074_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c54a0c1d-878d-4ead-828c-ad1f5f3335fd_isr_946074_39.stderr
parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486
parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: <AppFuture at 0x15230a018c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487
parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: <AppFuture at 0x15230a2ba090 state=pending>
parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: <AppFuture at 0x15230a183490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 489 has memoization hash dc3141fac5d4217b622c9277fc9b8e2c
parsl.dataflow.memoization INFO: Task 489 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/742ed215-3d88-4b43-bbd7-6780e63b3963_isr_946181_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/742ed215-3d88-4b43-bbd7-6780e63b3963_isr_946181_29.stderr
parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489
parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: <AppFuture at 0x15230a183210 state=pending>
parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490
parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: <AppFuture at 0x15230a180610 state=pending>
parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: <AppFuture at 0x15230a2f7fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 83d6d4d073e072bf776906b1fcc0f7a4
parsl.dataflow.memoization INFO: Task 492 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/3fd9931a-cf86-4779-ae42-8f9b9013dfc4_isr_946074_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/3fd9931a-cf86-4779-ae42-8f9b9013dfc4_isr_946074_51.stderr
parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492
parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: <AppFuture at 0x15230a2f57d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493
parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: <AppFuture at 0x15230a1cf350 state=pending>
parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: <AppFuture at 0x15230a004f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 538e56242c45457829bb80715cff9104
parsl.dataflow.memoization INFO: Task 495 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/82cdd83d-7712-4f80-a42a-2d58fa898785_isr_946180_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/82cdd83d-7712-4f80-a42a-2d58fa898785_isr_946180_18.stderr
parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495
parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: <AppFuture at 0x15230a1cef10 state=pending>
parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496
parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: <AppFuture at 0x15230a005ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: <AppFuture at 0x15230a005350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 498 has memoization hash a1502ef6109419472dd8e8e3f83ffb45
parsl.dataflow.memoization INFO: Task 498 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/1b6903d9-cad3-4b5b-8247-75bdc25180cb_isr_946180_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/1b6903d9-cad3-4b5b-8247-75bdc25180cb_isr_946180_22.stderr
parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498
parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: <AppFuture at 0x15230a005c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499
parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: <AppFuture at 0x15230a16b310 state=pending>
parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: <AppFuture at 0x15230a04ac50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 871ea64e2f8e468f328bdcdb779e7e04
parsl.dataflow.memoization INFO: Task 501 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7b2159e5-9437-4f1e-8faf-25e960a35a9c_isr_946181_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7b2159e5-9437-4f1e-8faf-25e960a35a9c_isr_946181_56.stderr
parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501
parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: <AppFuture at 0x15230a04b890 state=pending>
parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502
parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: <AppFuture at 0x15230a04b050 state=pending>
parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: <AppFuture at 0x15230a1baf50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 504 has memoization hash bfe9f0e9b44784c9b38dc1247497026d
parsl.dataflow.memoization INFO: Task 504 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bd8d946a-d286-4e80-a768-1de5a9834f7c_isr_946181_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bd8d946a-d286-4e80-a768-1de5a9834f7c_isr_946181_49.stderr
parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504
parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: <AppFuture at 0x15230a1b8b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505
parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: <AppFuture at 0x15230a1b8090 state=pending>
parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: <AppFuture at 0x15230a059d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 481347e9143b182fec7d5c5c2e9e9754
parsl.dataflow.memoization INFO: Task 507 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a27f0bd3-a495-4957-9dae-99815f6f90cc_isr_946182_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a27f0bd3-a495-4957-9dae-99815f6f90cc_isr_946182_10.stderr
parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507
parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: <AppFuture at 0x15230a05a5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508
parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: <AppFuture at 0x15230a059d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: <AppFuture at 0x15230a05bdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 406d05206a42a0160d341d0ca55e864e
parsl.dataflow.memoization INFO: Task 510 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/c11419ff-07c4-4f34-a216-34ccf5dcb1e1_isr_946073_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/c11419ff-07c4-4f34-a216-34ccf5dcb1e1_isr_946073_17.stderr
parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510
parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: <AppFuture at 0x15230a05acd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511
parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: <AppFuture at 0x15230a05b250 state=pending>
parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: <AppFuture at 0x15230a148bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 513 has memoization hash b0b79bea324c83022fc9a804fd3cd4e7
parsl.dataflow.memoization INFO: Task 513 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/53289755-253a-4d99-8a76-cf11866676d7_isr_946072_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/53289755-253a-4d99-8a76-cf11866676d7_isr_946072_46.stderr
parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513
parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: <AppFuture at 0x15230a14bfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514
parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: <AppFuture at 0x15230a05b050 state=pending>
parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: <AppFuture at 0x15230a199c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 0b1ff37add86bace2d26365df84ec7a4
parsl.dataflow.memoization INFO: Task 516 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/492e42be-127c-4459-98a8-6c1b3b94b0db_isr_946074_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/492e42be-127c-4459-98a8-6c1b3b94b0db_isr_946074_3.stderr
parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516
parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: <AppFuture at 0x15230a19ae90 state=pending>
parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517
parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: <AppFuture at 0x15230a199450 state=pending>
parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: <AppFuture at 0x15230a11db50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 519 has memoization hash bd411af703521dc4f5e54a8b7e0fb9b9
parsl.dataflow.memoization INFO: Task 519 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/634c0665-d966-45f3-8bf2-bb47e605940a_isr_946073_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/634c0665-d966-45f3-8bf2-bb47e605940a_isr_946073_24.stderr
parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519
parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: <AppFuture at 0x15230a149b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520
parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: <AppFuture at 0x15230a0605d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: <AppFuture at 0x15230a060a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 522 has memoization hash b82c4aa654529ffc8f31a1287f834874
parsl.dataflow.memoization INFO: Task 522 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3f30c75b-6ef4-4f41-84a3-7eb9a24989aa_isr_946181_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3f30c75b-6ef4-4f41-84a3-7eb9a24989aa_isr_946181_43.stderr
parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522
parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: <AppFuture at 0x15230a060790 state=pending>
parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523
parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: <AppFuture at 0x15230a061990 state=pending>
parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: <AppFuture at 0x15230a0230d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 90b33b2ce9a4b6765579ba19d3bffe2d
parsl.dataflow.memoization INFO: Task 525 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/55c10de7-c629-47d2-a312-c0c488048523_isr_946074_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/55c10de7-c629-47d2-a312-c0c488048523_isr_946074_60.stderr
parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525
parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: <AppFuture at 0x15230a060050 state=pending>
parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526
parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: <AppFuture at 0x15230a023d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: <AppFuture at 0x15230a021750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 50a36a0ed1cbd8e18a2338acc93bfbc0
parsl.dataflow.memoization INFO: Task 528 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dddc5323-cca0-42bc-8383-4b409a817dbd_isr_946073_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dddc5323-cca0-42bc-8383-4b409a817dbd_isr_946073_34.stderr
parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528
parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: <AppFuture at 0x15230a021bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529
parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: <AppFuture at 0x15230a021810 state=pending>
parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: <AppFuture at 0x15230a06b490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 390263a1ec7bc3ef44926d7642ae6270
parsl.dataflow.memoization INFO: Task 531 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b47f35a-5a8d-42de-91a1-dca4ad2c6cdb_isr_946182_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b47f35a-5a8d-42de-91a1-dca4ad2c6cdb_isr_946182_54.stderr
parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531
parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: <AppFuture at 0x15230a069990 state=pending>
parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532
parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: <AppFuture at 0x15230a0699d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: <AppFuture at 0x15230a06a1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 534 has memoization hash c6faa7c2974d146ddec5f59a36a2c10f
parsl.dataflow.memoization INFO: Task 534 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/97c84d08-d489-40ca-9f12-9bcd5e6f6564_isr_946074_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/97c84d08-d489-40ca-9f12-9bcd5e6f6564_isr_946074_41.stderr
parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534
parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: <AppFuture at 0x15230a06ad10 state=pending>
parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535
parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: <AppFuture at 0x15230a078a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: <AppFuture at 0x15230a0793d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 537 has memoization hash a8df1c43668d292740e05eb2c43a00dc
parsl.dataflow.memoization INFO: Task 537 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2f35693f-9d01-4cd5-893f-888200387d27_isr_946180_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2f35693f-9d01-4cd5-893f-888200387d27_isr_946180_28.stderr
parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537
parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: <AppFuture at 0x15230a079510 state=pending>
parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538
parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: <AppFuture at 0x15230a07a290 state=pending>
parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: <AppFuture at 0x15230a07a850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 540 has memoization hash a254a8cb7eed5dbce489be01f7b769ec
parsl.dataflow.memoization INFO: Task 540 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6858fbcf-1771-4358-bfe4-42d9ab39b902_isr_946072_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6858fbcf-1771-4358-bfe4-42d9ab39b902_isr_946072_25.stderr
parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540
parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: <AppFuture at 0x15230a07a510 state=pending>
parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541
parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: <AppFuture at 0x15230a07b8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: <AppFuture at 0x15230a07bcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 2537ba2f4ffaf05771ca766d50fb7a67
parsl.dataflow.memoization INFO: Task 543 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/967c872f-6755-477a-86d9-bfff50f3d7ea_isr_946180_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/967c872f-6755-477a-86d9-bfff50f3d7ea_isr_946180_1.stderr
parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543
parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: <AppFuture at 0x15230a07a490 state=pending>
parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544
parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: <AppFuture at 0x152308aa8410 state=pending>
parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: <AppFuture at 0x152308aa9090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 5f2cb482a5fcc013a6f40796888a1bb2
parsl.dataflow.memoization INFO: Task 546 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5cc460c4-2388-413c-8ea5-ecafdb7309ea_isr_946181_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5cc460c4-2388-413c-8ea5-ecafdb7309ea_isr_946181_38.stderr
parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546
parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: <AppFuture at 0x152308aa9210 state=pending>
parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547
parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: <AppFuture at 0x152308aaa0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: <AppFuture at 0x152308aaa410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 549 has memoization hash d441fdeef7f6d6407786c2017bbbaa24
parsl.dataflow.memoization INFO: Task 549 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2d08a6d2-02da-43de-9cb8-bd50a8cac4e8_isr_946181_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2d08a6d2-02da-43de-9cb8-bd50a8cac4e8_isr_946181_30.stderr
parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549
parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: <AppFuture at 0x152308aa9a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550
parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: <AppFuture at 0x152308aaae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: <AppFuture at 0x152308aab890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 2a4ef04b88cccd833f3c095275533d30
parsl.dataflow.memoization INFO: Task 552 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b7a2be9c-9960-4e3c-8dc9-3e043851a9d8_isr_946074_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b7a2be9c-9960-4e3c-8dc9-3e043851a9d8_isr_946074_22.stderr
parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552
parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: <AppFuture at 0x152308aabc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553
parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: <AppFuture at 0x152308ab80d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: <AppFuture at 0x152308ab8d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 7d7976fc5295946bfe0db465aff8d727
parsl.dataflow.memoization INFO: Task 555 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/246d67af-3ac9-4642-b567-0543388569cd_isr_946182_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/246d67af-3ac9-4642-b567-0543388569cd_isr_946182_47.stderr
parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555
parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: <AppFuture at 0x152308ab9610 state=pending>
parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556
parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: <AppFuture at 0x152308ab9a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: <AppFuture at 0x152308aba150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 558 has memoization hash acf49bce40428e68331df8c14c8a46e4
parsl.dataflow.memoization INFO: Task 558 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187
parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d695330f-ea22-40bb-9403-559643272de8_isr_946181_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d695330f-ea22-40bb-9403-559643272de8_isr_946181_33.stderr
parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558
parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: <AppFuture at 0x152308ab9050 state=pending>
parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559
parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: <AppFuture at 0x152308aba990 state=pending>
parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: <AppFuture at 0x152308abb590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 6e0172bc4bd0a06400afaee3f86c6de4
parsl.dataflow.memoization INFO: Task 561 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188
parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c05ea6f0-1961-4268-98d2-8d3419e5a0a6_isr_946074_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c05ea6f0-1961-4268-98d2-8d3419e5a0a6_isr_946074_30.stderr
parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561
parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: <AppFuture at 0x152308abb6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562
parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: <AppFuture at 0x152308abba50 state=pending>
parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: <AppFuture at 0x152308ac49d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 564 has memoization hash bce608f7636cd3284e527ca61b135955
parsl.dataflow.memoization INFO: Task 564 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189
parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/41ea4728-fa42-402f-89a7-d4bb200baa90_isr_946182_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/41ea4728-fa42-402f-89a7-d4bb200baa90_isr_946182_15.stderr
parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564
parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: <AppFuture at 0x152308ac5290 state=pending>
parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565
parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: <AppFuture at 0x152308ac5990 state=pending>
parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: <AppFuture at 0x152308ac5f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 8d48f50255138a4ee22f5b4744454701
parsl.dataflow.memoization INFO: Task 567 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190
parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ca554f25-3c78-44d8-996f-196beccd552a_isr_946072_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ca554f25-3c78-44d8-996f-196beccd552a_isr_946072_42.stderr
parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567
parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: <AppFuture at 0x152308ac56d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568
parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: <AppFuture at 0x152308ac6790 state=pending>
parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: <AppFuture at 0x152308ac7450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 9ac9a6659535123dbee0a3d69f6fa401
parsl.dataflow.memoization INFO: Task 570 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191
parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4c1c8608-0eb8-402e-b52a-7569f9e66aa2_isr_946074_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4c1c8608-0eb8-402e-b52a-7569f9e66aa2_isr_946074_16.stderr
parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570
parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: <AppFuture at 0x152308ac77d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571
parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: <AppFuture at 0x152308ac7990 state=pending>
parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: <AppFuture at 0x152308ad0810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 6bc40192f1ac31f64923b9fb733590a3
parsl.dataflow.memoization INFO: Task 573 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192
parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4cbdaa1e-ec19-408b-9f68-3930ff586fc0_isr_946072_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4cbdaa1e-ec19-408b-9f68-3930ff586fc0_isr_946072_59.stderr
parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573
parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: <AppFuture at 0x152308ad0550 state=pending>
parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574
parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: <AppFuture at 0x152308ad1250 state=pending>
parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: <AppFuture at 0x152308ad1c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 974deb334c6dbc6c8000df2c1362f17b
parsl.dataflow.memoization INFO: Task 576 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193
parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4ecd8c40-f41e-4ad3-a1d5-4333fa9b9d83_isr_946180_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4ecd8c40-f41e-4ad3-a1d5-4333fa9b9d83_isr_946180_27.stderr
parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576
parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: <AppFuture at 0x152308ad1890 state=pending>
parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577
parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: <AppFuture at 0x152308ad2a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: <AppFuture at 0x152308ad2fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 7310e2889a68806bb467f7421d4fa264
parsl.dataflow.memoization INFO: Task 579 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194
parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2633f997-3fd2-481c-b682-60f12d58fb5a_isr_946072_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2633f997-3fd2-481c-b682-60f12d58fb5a_isr_946072_34.stderr
parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579
parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: <AppFuture at 0x152308ad2610 state=pending>
parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580
parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: <AppFuture at 0x152308ad33d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: <AppFuture at 0x1523088d8410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 67da7f4e5bbdbee548349da449f3dca0
parsl.dataflow.memoization INFO: Task 582 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195
parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c9a103c5-417a-4687-8607-0130c19f87dd_isr_946182_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c9a103c5-417a-4687-8607-0130c19f87dd_isr_946182_33.stderr
parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582
parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: <AppFuture at 0x1523088d9050 state=pending>
parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583
parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: <AppFuture at 0x1523088d9390 state=pending>
parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: <AppFuture at 0x1523088d9850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 585 has memoization hash c79589490fcd02a30408341693c0788f
parsl.dataflow.memoization INFO: Task 585 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196
parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/83ce490a-11bb-4767-87d6-8aeed86070ef_isr_946073_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/83ce490a-11bb-4767-87d6-8aeed86070ef_isr_946073_51.stderr
parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585
parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: <AppFuture at 0x1523088d8bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586
parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: <AppFuture at 0x1523088d9f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: <AppFuture at 0x1523088dacd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 013ef9ce02bf75ac8d5c16699f952b8e
parsl.dataflow.memoization INFO: Task 588 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197
parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/a21c62c6-8b60-4aa5-8268-50c301fab6ff_isr_946072_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/a21c62c6-8b60-4aa5-8268-50c301fab6ff_isr_946072_39.stderr
parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588
parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: <AppFuture at 0x1523088db050 state=pending>
parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589
parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: <AppFuture at 0x1523088db510 state=pending>
parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: <AppFuture at 0x1523088dbe90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 80eecd97cf449ddb76fb7554606caf25
parsl.dataflow.memoization INFO: Task 591 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198
parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/605d5350-96d6-415a-a3c5-c5120346e167_isr_946181_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/605d5350-96d6-415a-a3c5-c5120346e167_isr_946181_10.stderr
parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591
parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: <AppFuture at 0x1523088dbd50 state=pending>
parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592
parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: <AppFuture at 0x1523088e8b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: <AppFuture at 0x1523088e9510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 594 has memoization hash eec59dde2a69ef53496cf201d9056c62
parsl.dataflow.memoization INFO: Task 594 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199
parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c97a06d4-a915-4c57-8c6d-690b7457596b_isr_946182_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c97a06d4-a915-4c57-8c6d-690b7457596b_isr_946182_18.stderr
parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594
parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: <AppFuture at 0x1523088e9250 state=pending>
parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595
parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: <AppFuture at 0x1523088e9cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: <AppFuture at 0x1523088ea8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 4d3f28db919a23283342df260782db4c
parsl.dataflow.memoization INFO: Task 597 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200
parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9de20dbe-4ed3-4fb1-9597-64b4791866af_isr_946074_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9de20dbe-4ed3-4fb1-9597-64b4791866af_isr_946074_40.stderr
parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597
parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: <AppFuture at 0x1523088e9ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598
parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: <AppFuture at 0x1523088eb0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: <AppFuture at 0x1523088ebd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 1438a622c22bcdfcfbbd643a4144f19b
parsl.dataflow.memoization INFO: Task 600 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201
parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/95774431-d4ad-45b3-83f1-3f752f27fea4_isr_946180_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/95774431-d4ad-45b3-83f1-3f752f27fea4_isr_946180_10.stderr
parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600
parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: <AppFuture at 0x1523088eb950 state=pending>
parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601
parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: <AppFuture at 0x1523088f8b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: <AppFuture at 0x1523088f9190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 5d120a66e19db6c5576a0e09b8bbd238
parsl.dataflow.memoization INFO: Task 603 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202
parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bd68158a-66e0-4235-a1a5-31fd9c0b9296_isr_946072_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bd68158a-66e0-4235-a1a5-31fd9c0b9296_isr_946072_23.stderr
parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603
parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: <AppFuture at 0x1523088f92d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604
parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: <AppFuture at 0x1523088fa050 state=pending>
parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: <AppFuture at 0x1523088fa690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 606 has memoization hash d15dab5043a70cb683f2aa443d9b35af
parsl.dataflow.memoization INFO: Task 606 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203
parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c51b1175-bf04-4e66-b0da-761164b00911_isr_946182_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c51b1175-bf04-4e66-b0da-761164b00911_isr_946182_45.stderr
parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606
parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: <AppFuture at 0x1523088fa7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607
parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: <AppFuture at 0x1523088fb510 state=pending>
parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: <AppFuture at 0x1523088fba50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 608904828061eb78457164dcd906fa4b
parsl.dataflow.memoization INFO: Task 609 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204
parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/dcb11e24-61b8-49cd-9be2-e66d25b2bdd8_isr_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/dcb11e24-61b8-49cd-9be2-e66d25b2bdd8_isr_946072_2.stderr
parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609
parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: <AppFuture at 0x1523088fbdd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610
parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: <AppFuture at 0x1523089009d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: <AppFuture at 0x152308900f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 840e9c7015e3dc9354b41a3ababe829d
parsl.dataflow.memoization INFO: Task 612 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205
parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58b3760b-b1b0-4aa2-a14e-9e6465bca753_isr_946182_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58b3760b-b1b0-4aa2-a14e-9e6465bca753_isr_946182_34.stderr
parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612
parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: <AppFuture at 0x1523089003d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613
parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: <AppFuture at 0x152308901f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: <AppFuture at 0x152308902290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 615 has memoization hash d23bb6260d4784799e3512038c2c31a5
parsl.dataflow.memoization INFO: Task 615 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206
parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9a7d1483-ae0b-422a-9430-f5a9de782b05_isr_946073_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9a7d1483-ae0b-422a-9430-f5a9de782b05_isr_946073_47.stderr
parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615
parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: <AppFuture at 0x152308902950 state=pending>
parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616
parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: <AppFuture at 0x152308902b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: <AppFuture at 0x1523089036d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 16dfd65255226c6d6fb231e0fae67032
parsl.dataflow.memoization INFO: Task 618 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207
parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6b8ab4b1-64f3-48da-a327-a1a1e0eb059c_isr_946182_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6b8ab4b1-64f3-48da-a327-a1a1e0eb059c_isr_946182_8.stderr
parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618
parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: <AppFuture at 0x152308903d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619
parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: <AppFuture at 0x152308903bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: <AppFuture at 0x152308908a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 6bd265e76daad9ddd5071103ef750b00
parsl.dataflow.memoization INFO: Task 621 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088edda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208
parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/aaae5965-a612-4dd8-8f81-daa8a835b5a0_isr_946072_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/aaae5965-a612-4dd8-8f81-daa8a835b5a0_isr_946072_5.stderr
parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621
parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: <AppFuture at 0x152308909050 state=pending>
parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622
parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: <AppFuture at 0x1523089094d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: <AppFuture at 0x152308909e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 21c4fb80d598b47efc545881b157837f
parsl.dataflow.memoization INFO: Task 624 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209
parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/163ceeda-0e88-4f4a-a2af-6289bc324a39_isr_946072_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/163ceeda-0e88-4f4a-a2af-6289bc324a39_isr_946072_36.stderr
parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624
parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: <AppFuture at 0x152308909bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625
parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: <AppFuture at 0x15230890a3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: <AppFuture at 0x15230890b290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 0efeb25e915a0e789f636752b88fab13
parsl.dataflow.memoization INFO: Task 627 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210
parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/68b2953c-da29-4d18-854a-a9b534772bb8_isr_946181_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/68b2953c-da29-4d18-854a-a9b534772bb8_isr_946181_39.stderr
parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627
parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: <AppFuture at 0x15230890a7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628
parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: <AppFuture at 0x152308900310 state=pending>
parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: <AppFuture at 0x152308910710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 895a662cc9ba6a597a1bd8604dc6a498
parsl.dataflow.memoization INFO: Task 630 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211
parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8ce900f6-829c-465d-b474-1ac34ae6098e_isr_946072_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8ce900f6-829c-465d-b474-1ac34ae6098e_isr_946072_32.stderr
parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630
parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: <AppFuture at 0x152308910d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631
parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: <AppFuture at 0x152308910410 state=pending>
parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: <AppFuture at 0x152308911a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 633 has memoization hash bbce37f623b23accf9563c03a791d941
parsl.dataflow.memoization INFO: Task 633 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212
parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a336202d-ae2f-4ec9-9a68-d43dc46a262a_isr_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a336202d-ae2f-4ec9-9a68-d43dc46a262a_isr_946074_2.stderr
parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633
parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: <AppFuture at 0x152308910f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634
parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: <AppFuture at 0x1523089122d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: <AppFuture at 0x152308912ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 31c61a2b8d90f93c435f99dc834489e2
parsl.dataflow.memoization INFO: Task 636 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213
parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/da95c5f4-c81a-4894-bcbd-6549bbaa07c7_isr_946073_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/da95c5f4-c81a-4894-bcbd-6549bbaa07c7_isr_946073_32.stderr
parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636
parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: <AppFuture at 0x152308913cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637
parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: <AppFuture at 0x1523089137d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: <AppFuture at 0x15230891c390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 639 has memoization hash e55de0c29287e09958c035a881982aee
parsl.dataflow.memoization INFO: Task 639 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214
parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a35dd479-e487-4c39-a391-1234c20ebab2_isr_946182_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a35dd479-e487-4c39-a391-1234c20ebab2_isr_946182_12.stderr
parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639
parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: <AppFuture at 0x1523089136d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640
parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: <AppFuture at 0x15230891cb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: <AppFuture at 0x15230891d750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 7b1e448b14f9730ac60216153b689e34
parsl.dataflow.memoization INFO: Task 642 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215
parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/609434ca-8ddd-4d6f-b373-e18db27e1923_isr_946073_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/609434ca-8ddd-4d6f-b373-e18db27e1923_isr_946073_29.stderr
parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642
parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: <AppFuture at 0x15230891cb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643
parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: <AppFuture at 0x15230891e650 state=pending>
parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: <AppFuture at 0x15230891ec10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 09a9f10336834680afc120fba473b8a3
parsl.dataflow.memoization INFO: Task 645 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216
parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9cc70ad5-bf38-4758-b160-78b6fcca41ce_isr_946182_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9cc70ad5-bf38-4758-b160-78b6fcca41ce_isr_946182_32.stderr
parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645
parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: <AppFuture at 0x15230891e690 state=pending>
parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646
parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: <AppFuture at 0x15230891f250 state=pending>
parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: <AppFuture at 0x15230891ffd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 648 has memoization hash cf90a58e215005e43870e4396025195b
parsl.dataflow.memoization INFO: Task 648 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217
parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/23e248f1-511f-41e3-bee7-c8c493e4ed95_isr_946072_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/23e248f1-511f-41e3-bee7-c8c493e4ed95_isr_946072_1.stderr
parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648
parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: <AppFuture at 0x15230891fc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649
parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: <AppFuture at 0x152308928fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: <AppFuture at 0x1523089294d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 651 has memoization hash bc1165301391547fcfe1be26b225581a
parsl.dataflow.memoization INFO: Task 651 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088efce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218
parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/79f3f393-2cf0-4267-a1cf-8713afb81963_isr_946074_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/79f3f393-2cf0-4267-a1cf-8713afb81963_isr_946074_59.stderr
parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651
parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: <AppFuture at 0x15231b270790 state=pending>
parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652
parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: <AppFuture at 0x15230892a310 state=pending>
parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: <AppFuture at 0x15230892a910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 4f55910e9817beba58298d326c387890
parsl.dataflow.memoization INFO: Task 654 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219
parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b164f24-c574-4dc8-88bb-df21f6a67e8b_isr_946180_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b164f24-c574-4dc8-88bb-df21f6a67e8b_isr_946180_21.stderr
parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654
parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: <AppFuture at 0x15230892a650 state=pending>
parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655
parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: <AppFuture at 0x15230892b7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: <AppFuture at 0x15230892bd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 8de02a3a3589ba28a1368bfc4576c5cf
parsl.dataflow.memoization INFO: Task 657 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220
parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a2c01222-6a7c-42d0-9b5d-255c8495a630_isr_946180_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a2c01222-6a7c-42d0-9b5d-255c8495a630_isr_946180_40.stderr
parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657
parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: <AppFuture at 0x15230a24c410 state=pending>
parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658
parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: <AppFuture at 0x15230893c190 state=pending>
parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: <AppFuture at 0x15230893d110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 369ba38c1db95e03e2f408e6ce4986e8
parsl.dataflow.memoization INFO: Task 660 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221
parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/fd4ffa07-dfc7-4d6a-b3bc-ef71a100b94b_isr_946073_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/fd4ffa07-dfc7-4d6a-b3bc-ef71a100b94b_isr_946073_8.stderr
parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660
parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: <AppFuture at 0x15230893c550 state=pending>
parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661
parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: <AppFuture at 0x15230893ded0 state=pending>
parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: <AppFuture at 0x15230893e510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 301442581eb4289cee93a54747d9bd63
parsl.dataflow.memoization INFO: Task 663 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222
parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/03c352fd-0a18-43fa-bd0d-d4d5d9d50d0a_isr_946180_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/03c352fd-0a18-43fa-bd0d-d4d5d9d50d0a_isr_946180_11.stderr
parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663
parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: <AppFuture at 0x15230893e250 state=pending>
parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664
parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: <AppFuture at 0x15230893edd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: <AppFuture at 0x15230893fa10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 666 has memoization hash ce69257568bfe5f5fb9eefdfd4bf1d66
parsl.dataflow.memoization INFO: Task 666 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223
parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e30c92df-d569-4cb7-9198-320c124edeef_isr_946072_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e30c92df-d569-4cb7-9198-320c124edeef_isr_946072_17.stderr
parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666
parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: <AppFuture at 0x15230893ff50 state=pending>
parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667
parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: <AppFuture at 0x152308940650 state=pending>
parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: <AppFuture at 0x152308940ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 669 has memoization hash d2d086b3b9ac785839d4077400d4110c
parsl.dataflow.memoization INFO: Task 669 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224
parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2492a8e2-3b55-47d1-8d7c-45ef9b871653_isr_946182_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2492a8e2-3b55-47d1-8d7c-45ef9b871653_isr_946182_58.stderr
parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669
parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: <AppFuture at 0x152308940b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670
parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: <AppFuture at 0x1523089416d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: <AppFuture at 0x152308942210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 7ed5c0a1b193a7cdca97a5dea2f141a2
parsl.dataflow.memoization INFO: Task 672 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225
parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c483220b-2763-4ada-ae7f-ee9aeaab9de3_isr_946072_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c483220b-2763-4ada-ae7f-ee9aeaab9de3_isr_946072_43.stderr
parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672
parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: <AppFuture at 0x152308942950 state=pending>
parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673
parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: <AppFuture at 0x152308942b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: <AppFuture at 0x152308943550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 675 has memoization hash b9001c18ad2a10e39bd49b2cdab7ecaa
parsl.dataflow.memoization INFO: Task 675 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226
parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/0d2b5a11-d143-4aa9-80ea-2a3a439ee7a1_isr_946074_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/0d2b5a11-d143-4aa9-80ea-2a3a439ee7a1_isr_946074_1.stderr
parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675
parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: <AppFuture at 0x152308943e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676
parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: <AppFuture at 0x15230890bcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: <AppFuture at 0x1523089509d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 678 has memoization hash da8e8210aa6a4585112919da5b39795f
parsl.dataflow.memoization INFO: Task 678 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227
parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/066c93c4-919b-43ac-a81d-f2d3a3a3bf8f_isr_946072_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/066c93c4-919b-43ac-a81d-f2d3a3a3bf8f_isr_946072_4.stderr
parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678
parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: <AppFuture at 0x152308951050 state=pending>
parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679
parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: <AppFuture at 0x1523089510d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: <AppFuture at 0x152308951d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 681 has memoization hash c83ea403ad92304a6f649c1ee5ea62d5
parsl.dataflow.memoization INFO: Task 681 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228
parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fd2abaed-e0b2-43f5-b21c-ec68af16d511_isr_946072_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fd2abaed-e0b2-43f5-b21c-ec68af16d511_isr_946072_22.stderr
parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681
parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: <AppFuture at 0x152308951250 state=pending>
parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682
parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: <AppFuture at 0x152308952610 state=pending>
parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: <AppFuture at 0x152308953210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 47943a83766f6bcf8f8a3d79d7b5bf5b
parsl.dataflow.memoization INFO: Task 684 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229
parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/07dd00d6-f267-45ce-9be9-960729ce0e87_isr_946072_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/07dd00d6-f267-45ce-9be9-960729ce0e87_isr_946072_27.stderr
parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684
parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: <AppFuture at 0x152308953590 state=pending>
parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685
parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: <AppFuture at 0x15230890bad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: <AppFuture at 0x152308954690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 67548aad3e6901d89c09b292a746849e
parsl.dataflow.memoization INFO: Task 687 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230
parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e6bae503-d94a-4baa-aecc-cc14e43c976f_isr_946180_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e6bae503-d94a-4baa-aecc-cc14e43c976f_isr_946180_26.stderr
parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687
parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: <AppFuture at 0x152308953ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688
parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: <AppFuture at 0x1523089552d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: <AppFuture at 0x152308955a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 690 has memoization hash dede1e74e2072beb53e6062619c3f880
parsl.dataflow.memoization INFO: Task 690 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231
parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/224a8166-0ac3-4b97-b057-8c3316fb0038_isr_946073_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/224a8166-0ac3-4b97-b057-8c3316fb0038_isr_946073_59.stderr
parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690
parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: <AppFuture at 0x1523089560d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691
parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: <AppFuture at 0x152308955f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: <AppFuture at 0x152308956ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 3c0535647ea9c3fad4179a5103e56884
parsl.dataflow.memoization INFO: Task 693 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232
parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/5e0a1768-9a78-41d3-997f-b66afd470e53_isr_946072_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/5e0a1768-9a78-41d3-997f-b66afd470e53_isr_946072_9.stderr
parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693
parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: <AppFuture at 0x152308957050 state=pending>
parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694
parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: <AppFuture at 0x1523089578d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: <AppFuture at 0x152308960310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 696 has memoization hash e556b8d469fa3fc95edb03801b8a9f37
parsl.dataflow.memoization INFO: Task 696 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233
parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3daa38cd-cd51-4c14-a606-79aaa3186454_isr_946182_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3daa38cd-cd51-4c14-a606-79aaa3186454_isr_946182_20.stderr
parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696
parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: <AppFuture at 0x1523089575d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697
parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: <AppFuture at 0x1523089611d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: <AppFuture at 0x1523089616d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 28301de5ba44fafb42b4dea6cb814f0c
parsl.dataflow.memoization INFO: Task 699 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234
parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2417af8f-e939-45d0-8cce-8fe9cd269686_isr_946182_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2417af8f-e939-45d0-8cce-8fe9cd269686_isr_946182_6.stderr
parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699
parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: <AppFuture at 0x152308960ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700
parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: <AppFuture at 0x152308962350 state=pending>
parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: <AppFuture at 0x152308962b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 702 has memoization hash d7202cc674f520cebffb13479b97804d
parsl.dataflow.memoization INFO: Task 702 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235
parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/01aafa68-81fa-4c6d-a827-ba9743a461a9_isr_946182_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/01aafa68-81fa-4c6d-a827-ba9743a461a9_isr_946182_57.stderr
parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702
parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: <AppFuture at 0x152308960a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703
parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: <AppFuture at 0x1523089639d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: <AppFuture at 0x152308963f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 705 has memoization hash ba0c2d02953fec878d959ca11dcea4a6
parsl.dataflow.memoization INFO: Task 705 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236
parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c5f11ee4-cc6c-462f-9a6b-9fea5d78bb69_isr_946074_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c5f11ee4-cc6c-462f-9a6b-9fea5d78bb69_isr_946074_34.stderr
parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705
parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: <AppFuture at 0x152308963c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706
parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: <AppFuture at 0x15230896ce50 state=pending>
parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: <AppFuture at 0x15230896d390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 1bcf9ec4d443b949b2c496c3288828cd
parsl.dataflow.memoization INFO: Task 708 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237
parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9c7ce740-8655-47dd-8d57-a2be38fb8624_isr_946180_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9c7ce740-8655-47dd-8d57-a2be38fb8624_isr_946180_4.stderr
parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708
parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: <AppFuture at 0x15230896c890 state=pending>
parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709
parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: <AppFuture at 0x15230896e5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: <AppFuture at 0x15230896e9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 56f26d8b6d780b48daa5a7074f94eb33
parsl.dataflow.memoization INFO: Task 711 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238
parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/14719507-defe-4a2a-af61-a98b0d450bea_isr_946074_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/14719507-defe-4a2a-af61-a98b0d450bea_isr_946074_21.stderr
parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711
parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: <AppFuture at 0x15230896de50 state=pending>
parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712
parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: <AppFuture at 0x15230896f8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: <AppFuture at 0x15230896fdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 646f4670f4704516ab9a46ae362748db
parsl.dataflow.memoization INFO: Task 714 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239
parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47ef7f8f-4eb0-4836-a5ed-3b4f29097b82_isr_946073_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47ef7f8f-4eb0-4836-a5ed-3b4f29097b82_isr_946073_3.stderr
parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714
parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: <AppFuture at 0x15230896fb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715
parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: <AppFuture at 0x152308978650 state=pending>
parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: <AppFuture at 0x1523089791d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 08396f2a2e5e6782bae39ccf9979455f
parsl.dataflow.memoization INFO: Task 717 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240
parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4df5c816-e20b-4e8d-8651-be1a740398d9_isr_946072_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4df5c816-e20b-4e8d-8651-be1a740398d9_isr_946072_62.stderr
parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717
parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: <AppFuture at 0x15230892a2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718
parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: <AppFuture at 0x152308979c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: <AppFuture at 0x15230897a650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 9adedb991c88c93a78742f39d4a2cbe2
parsl.dataflow.memoization INFO: Task 720 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241
parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dbb60a14-2224-459a-ab6c-1521ab61b068_isr_946073_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dbb60a14-2224-459a-ab6c-1521ab61b068_isr_946073_13.stderr
parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720
parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: <AppFuture at 0x15230897b610 state=pending>
parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721
parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: <AppFuture at 0x15230897b490 state=pending>
parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: <AppFuture at 0x15230897ba10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 723 has memoization hash bac5ff2f5625a154716176d18cb769c2
parsl.dataflow.memoization INFO: Task 723 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242
parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bb5105c1-7f1a-43f0-a673-c9dd7cc695c0_isr_946181_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bb5105c1-7f1a-43f0-a673-c9dd7cc695c0_isr_946181_18.stderr
parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723
parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: <AppFuture at 0x15230897be10 state=pending>
parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724
parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: <AppFuture at 0x1523089806d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: <AppFuture at 0x152308980f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 726 has memoization hash f9998b46f363aad361bb789a0e5d2ca0
parsl.dataflow.memoization INFO: Task 726 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243
parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2539d1b2-734f-4a51-ab2f-6c07f81422f2_isr_946180_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2539d1b2-734f-4a51-ab2f-6c07f81422f2_isr_946180_36.stderr
parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726
parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: <AppFuture at 0x15230897ad50 state=pending>
parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727
parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: <AppFuture at 0x152308981650 state=pending>
parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: <AppFuture at 0x152308982350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 96a6309014ae3c639538f2dc279d119a
parsl.dataflow.memoization INFO: Task 729 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244
parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1cdf86b6-ce26-493c-a2a0-021fe3b13b9d_isr_946182_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1cdf86b6-ce26-493c-a2a0-021fe3b13b9d_isr_946182_25.stderr
parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729
parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: <AppFuture at 0x152308983310 state=pending>
parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730
parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: <AppFuture at 0x152308982ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: <AppFuture at 0x152308983710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 9fa2703b8540ccd69cb871a2a791977c
parsl.dataflow.memoization INFO: Task 732 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245
parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fe568b7b-fc62-4ad1-b3aa-3688d8db36c1_isr_946180_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fe568b7b-fc62-4ad1-b3aa-3688d8db36c1_isr_946180_41.stderr
parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732
parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: <AppFuture at 0x152308983fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733
parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: <AppFuture at 0x152308990110 state=pending>
parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: <AppFuture at 0x152308990b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 0a38de3fd74a31936ccbc9f66b7d8963
parsl.dataflow.memoization INFO: Task 735 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246
parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/82b3c135-a00a-4608-be57-1d33cffdad2b_isr_946072_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/82b3c135-a00a-4608-be57-1d33cffdad2b_isr_946072_30.stderr
parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735
parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: <AppFuture at 0x152308990690 state=pending>
parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736
parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: <AppFuture at 0x152308991590 state=pending>
parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: <AppFuture at 0x152308991f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 738 has memoization hash cb58e6a41da4011841ad156dafd133a7
parsl.dataflow.memoization INFO: Task 738 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247
parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23152ea4-930c-4120-8cd4-8d68ad1b0c8d_isr_946073_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23152ea4-930c-4120-8cd4-8d68ad1b0c8d_isr_946073_7.stderr
parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738
parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: <AppFuture at 0x152308991c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739
parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: <AppFuture at 0x152308992410 state=pending>
parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: <AppFuture at 0x152308993350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 741 has memoization hash fc3451326df6ad122539a6be7e045fbd
parsl.dataflow.memoization INFO: Task 741 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248
parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/bcab3b52-02cb-4bc8-aed0-1025445fdf27_isr_946074_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/bcab3b52-02cb-4bc8-aed0-1025445fdf27_isr_946074_18.stderr
parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741
parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: <AppFuture at 0x1523089934d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742
parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: <AppFuture at 0x152308993850 state=pending>
parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: <AppFuture at 0x152308998790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 744 has memoization hash e9f090ae29978f6b05ae4a09ecbd002c
parsl.dataflow.memoization INFO: Task 744 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249
parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3733df0e-3ba4-41ac-b9b6-d8fd427ca02c_isr_946180_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3733df0e-3ba4-41ac-b9b6-d8fd427ca02c_isr_946180_34.stderr
parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744
parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: <AppFuture at 0x152308998bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745
parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: <AppFuture at 0x152308998410 state=pending>
parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: <AppFuture at 0x152308999b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 2ba7d608442d1218ec02f2369e5ec1d9
parsl.dataflow.memoization INFO: Task 747 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250
parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/febfb9df-8a17-4dfc-957f-5638ed5d4d7d_isr_946181_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/febfb9df-8a17-4dfc-957f-5638ed5d4d7d_isr_946181_20.stderr
parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747
parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: <AppFuture at 0x152308999010 state=pending>
parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748
parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: <AppFuture at 0x15230899a450 state=pending>
parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: <AppFuture at 0x15230899af50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 85ab9f34485fd6f3fe1cfa7fe610c145
parsl.dataflow.memoization INFO: Task 750 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251
parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e2b58cb3-4588-4a97-bf51-1d7f2bc8d8ae_isr_946074_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e2b58cb3-4588-4a97-bf51-1d7f2bc8d8ae_isr_946074_49.stderr
parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750
parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: <AppFuture at 0x15230899bd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751
parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: <AppFuture at 0x15230899b410 state=pending>
parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: <AppFuture at 0x1523089a0450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 753 has memoization hash cb4b9e6fe72e24102589ab602b0d127c
parsl.dataflow.memoization INFO: Task 753 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252
parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82f91d2f-263e-4f1b-8661-9d9f5a9104ac_isr_946181_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82f91d2f-263e-4f1b-8661-9d9f5a9104ac_isr_946181_1.stderr
parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753
parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: <AppFuture at 0x1523089a0d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754
parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: <AppFuture at 0x1523089a11d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: <AppFuture at 0x1523089a1750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 519f425da493c92364c0eabb2baaddcd
parsl.dataflow.memoization INFO: Task 756 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253
parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/acf64e70-5917-4f94-abc9-9ddbeddcb0d9_isr_946181_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/acf64e70-5917-4f94-abc9-9ddbeddcb0d9_isr_946181_47.stderr
parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756
parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: <AppFuture at 0x152308981f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757
parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: <AppFuture at 0x1523089a1f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: <AppFuture at 0x1523089a2b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 04f6a236dbc3c1c3c8d4d6b62d9f0219
parsl.dataflow.memoization INFO: Task 759 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254
parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/bc0ee1a7-576d-4f35-b49b-694a1fc5fbce_isr_946182_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/bc0ee1a7-576d-4f35-b49b-694a1fc5fbce_isr_946182_52.stderr
parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759
parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: <AppFuture at 0x1523089a3b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760
parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: <AppFuture at 0x1523089a3a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: <AppFuture at 0x1523089a3f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 762 has memoization hash c9e8f54b31619552c61ec599a7095914
parsl.dataflow.memoization INFO: Task 762 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255
parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/20838f3f-cd5a-4478-b5b7-2b31399e01df_isr_946073_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/20838f3f-cd5a-4478-b5b7-2b31399e01df_isr_946073_38.stderr
parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762
parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: <AppFuture at 0x1523089634d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763
parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: <AppFuture at 0x1523089b4750 state=pending>
parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: <AppFuture at 0x1523089b5410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 433ba0f5dee72611eed2bbdd9a440cfa
parsl.dataflow.memoization INFO: Task 765 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256
parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/4ff20f97-6e57-49b7-9643-d37f637c77dc_isr_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/4ff20f97-6e57-49b7-9643-d37f637c77dc_isr_946182_31.stderr
parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765
parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: <AppFuture at 0x1523089b4a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766
parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: <AppFuture at 0x1523089b5d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: <AppFuture at 0x1523089b6950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 768 has memoization hash ea2c97b006e3ad63325cdd000b3a2f2d
parsl.dataflow.memoization INFO: Task 768 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257
parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/36c820e3-95c0-47a8-87e6-3747b5370860_isr_946073_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/36c820e3-95c0-47a8-87e6-3747b5370860_isr_946073_41.stderr
parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768
parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: <AppFuture at 0x1523089b6690 state=pending>
parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769
parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: <AppFuture at 0x1523089b7710 state=pending>
parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: <AppFuture at 0x1523089b7cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 246d6c3f2ebe7d9c83e2931bb617226d
parsl.dataflow.memoization INFO: Task 771 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258
parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d5f13ff1-aeb6-4fb7-977f-5f082e629fbf_isr_946073_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d5f13ff1-aeb6-4fb7-977f-5f082e629fbf_isr_946073_58.stderr
parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771
parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: <AppFuture at 0x1523089b79d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772
parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: <AppFuture at 0x1523089b84d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: <AppFuture at 0x1523089b9090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 2bdebc7855a2e86382bb82b3899dacd4
parsl.dataflow.memoization INFO: Task 774 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259
parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/63b0f786-00f7-456a-8d62-41433795da49_isr_946181_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/63b0f786-00f7-456a-8d62-41433795da49_isr_946181_45.stderr
parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774
parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: <AppFuture at 0x1523089a1f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775
parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: <AppFuture at 0x1523089b9990 state=pending>
parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: <AppFuture at 0x1523089ba4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 071baf99a9a45ae57e840e8f69b4865c
parsl.dataflow.memoization INFO: Task 777 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260
parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2c60f8fa-1306-45eb-a26c-8484da33ce51_isr_946072_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2c60f8fa-1306-45eb-a26c-8484da33ce51_isr_946072_58.stderr
parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777
parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: <AppFuture at 0x1523089b98d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778
parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: <AppFuture at 0x1523089bad50 state=pending>
parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: <AppFuture at 0x1523089bb890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 780 has memoization hash f7bf4fc18247a96671b1f6b4d075e86f
parsl.dataflow.memoization INFO: Task 780 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261
parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/93f5a6b4-2662-44b9-a711-6864b00c37c8_isr_946073_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/93f5a6b4-2662-44b9-a711-6864b00c37c8_isr_946073_46.stderr
parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780
parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: <AppFuture at 0x1523089bbc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781
parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: <AppFuture at 0x1523089c87d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: <AppFuture at 0x1523089c8d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 783 has memoization hash b00f4c886bdc707ad21c1092a6427e35
parsl.dataflow.memoization INFO: Task 783 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262
parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/442760e8-47e6-480f-8c4d-3d4c20cf9d2e_isr_946181_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/442760e8-47e6-480f-8c4d-3d4c20cf9d2e_isr_946181_59.stderr
parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783
parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: <AppFuture at 0x1523089c85d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784
parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: <AppFuture at 0x1523089c9bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: <AppFuture at 0x1523089ca110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 786 has memoization hash db0401fdb0c5ab5f159faef5c65ebd21
parsl.dataflow.memoization INFO: Task 786 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263
parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0858d0f1-8788-498a-bc3d-e88ccf6c9163_isr_946073_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0858d0f1-8788-498a-bc3d-e88ccf6c9163_isr_946073_33.stderr
parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786
parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: <AppFuture at 0x1523089ca290 state=pending>
parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787
parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: <AppFuture at 0x1523089ca8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: <AppFuture at 0x1523089cb510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 1aadc87a5ad40262ae240310e8292da3
parsl.dataflow.memoization INFO: Task 789 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264
parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9987bd13-97b4-4e56-a8b5-72d51256b2ea_isr_946074_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9987bd13-97b4-4e56-a8b5-72d51256b2ea_isr_946074_15.stderr
parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789
parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: <AppFuture at 0x1523089cb910 state=pending>
parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790
parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: <AppFuture at 0x1523089babd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: <AppFuture at 0x1523089d0950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 13287307bf373c551e5f9e9301bc47db
parsl.dataflow.memoization INFO: Task 792 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265
parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23de528a-7fb5-4b85-96e0-059139828192_isr_946073_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23de528a-7fb5-4b85-96e0-059139828192_isr_946073_40.stderr
parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792
parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: <AppFuture at 0x152308979950 state=pending>
parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793
parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: <AppFuture at 0x1523089d1350 state=pending>
parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: <AppFuture at 0x1523089d1c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 795 has memoization hash aafa9eda03f333755f7aa55e0f75ddcd
parsl.dataflow.memoization INFO: Task 795 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266
parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/63fd7a00-b843-4152-9241-8af230cfa323_isr_946073_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/63fd7a00-b843-4152-9241-8af230cfa323_isr_946073_55.stderr
parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795
parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: <AppFuture at 0x1523089d25d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796
parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: <AppFuture at 0x1523089d2590 state=pending>
parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: <AppFuture at 0x1523089d30d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 19b298f603a99a1c209383397e813232
parsl.dataflow.memoization INFO: Task 798 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267
parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0332f59e-aad6-4fa2-9734-b1ee2932f0c2_isr_946073_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0332f59e-aad6-4fa2-9734-b1ee2932f0c2_isr_946073_30.stderr
parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798
parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: <AppFuture at 0x1523089d2d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799
parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: <AppFuture at 0x1523089c92d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: <AppFuture at 0x1523087dc510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 801 has memoization hash f69b06116efd836ab85c64cf5da34f38
parsl.dataflow.memoization INFO: Task 801 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268
parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7c955426-e7f7-427c-af98-37d9d79d69d9_isr_946180_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7c955426-e7f7-427c-af98-37d9d79d69d9_isr_946180_13.stderr
parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801
parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: <AppFuture at 0x1523087dc690 state=pending>
parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802
parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: <AppFuture at 0x1523087dcd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: <AppFuture at 0x1523087dd9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 37a90844a4b1a600c02ab7dfbfb0199f
parsl.dataflow.memoization INFO: Task 804 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269
parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9dfac037-cc93-481e-be66-300019aba4c3_isr_946073_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9dfac037-cc93-481e-be66-300019aba4c3_isr_946073_37.stderr
parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804
parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: <AppFuture at 0x1523087dd1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805
parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: <AppFuture at 0x1523087de290 state=pending>
parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: <AppFuture at 0x1523087dedd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 807 has memoization hash c21c39c0ac33c18893d9d20cdf0d6c47
parsl.dataflow.memoization INFO: Task 807 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270
parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/62614a60-f1c5-490f-9efa-76c10957854e_isr_946073_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/62614a60-f1c5-490f-9efa-76c10957854e_isr_946073_42.stderr
parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807
parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: <AppFuture at 0x1523087deb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808
parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: <AppFuture at 0x1523087df9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: <AppFuture at 0x1523087e8110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 2e059761e5407ec7854ccc0d9ae3f3d6
parsl.dataflow.memoization INFO: Task 810 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271
parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/dead9ac2-b5d5-4ce1-bfb9-28acc89d66a5_isr_946180_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/dead9ac2-b5d5-4ce1-bfb9-28acc89d66a5_isr_946180_15.stderr
parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810
parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: <AppFuture at 0x1523087dfe90 state=pending>
parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811
parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: <AppFuture at 0x1523087e9090 state=pending>
parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: <AppFuture at 0x1523087e9610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 81324246537705f5e01cf1d033d696de
parsl.dataflow.memoization INFO: Task 813 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272
parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fe86de84-b263-468c-8628-16103509a959_isr_946074_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fe86de84-b263-468c-8628-16103509a959_isr_946074_35.stderr
parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813
parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: <AppFuture at 0x1523087e8990 state=pending>
parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814
parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: <AppFuture at 0x1523087ea590 state=pending>
parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: <AppFuture at 0x1523087eaa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 816 has memoization hash d1469e11fdbc3572ebb4c8dfd22e52be
parsl.dataflow.memoization INFO: Task 816 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273
parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ddb71e9a-e890-4098-a4d0-cbcc37f7f905_isr_946073_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ddb71e9a-e890-4098-a4d0-cbcc37f7f905_isr_946073_9.stderr
parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816
parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: <AppFuture at 0x1523087e9e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817
parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: <AppFuture at 0x1523087eb910 state=pending>
parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: <AppFuture at 0x1523087ebed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 819 has memoization hash bcfa7983bf15c0c74a1102d779744a5b
parsl.dataflow.memoization INFO: Task 819 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274
parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/25694540-773d-4240-b83b-7ca2ca26b401_isr_946074_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/25694540-773d-4240-b83b-7ca2ca26b401_isr_946074_27.stderr
parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819
parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: <AppFuture at 0x1523087ebb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820
parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: <AppFuture at 0x1523087f0f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: <AppFuture at 0x1523087f1250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 822 has memoization hash bc1f23a721bce796250c7db6939c89a3
parsl.dataflow.memoization INFO: Task 822 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275
parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/88f6d1a2-46e0-4868-abe6-c8f965364ab2_isr_946180_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/88f6d1a2-46e0-4868-abe6-c8f965364ab2_isr_946180_47.stderr
parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822
parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: <AppFuture at 0x1523087f19d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823
parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: <AppFuture at 0x1523087f1a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: <AppFuture at 0x1523087f2690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 825 has memoization hash cd3bc14476af18f74f644fa50e6205d8
parsl.dataflow.memoization INFO: Task 825 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276
parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bc01ed06-0486-46fc-9656-8f539d3c460a_isr_946181_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bc01ed06-0486-46fc-9656-8f539d3c460a_isr_946181_19.stderr
parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825
parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: <AppFuture at 0x1523087f2810 state=pending>
parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826
parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: <AppFuture at 0x1523087f2dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: <AppFuture at 0x1523087f3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 828 has memoization hash ed43a283fe72594061998ddfee93d363
parsl.dataflow.memoization INFO: Task 828 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277
parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e37df7a0-b464-4031-8c26-a2712f9b4c5a_isr_946180_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e37df7a0-b464-4031-8c26-a2712f9b4c5a_isr_946180_50.stderr
parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828
parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: <AppFuture at 0x1523087f3650 state=pending>
parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829
parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: <AppFuture at 0x1523088008d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: <AppFuture at 0x152308800ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 831 has memoization hash 24be507e6ed1072ff8ce777720ffc72f
parsl.dataflow.memoization INFO: Task 831 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278
parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ccb02535-7def-45af-9a4e-2e794439e079_isr_946182_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ccb02535-7def-45af-9a4e-2e794439e079_isr_946182_1.stderr
parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831
parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: <AppFuture at 0x152308801f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832
parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: <AppFuture at 0x152308801390 state=pending>
parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: <AppFuture at 0x1523088023d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 834 has memoization hash 9b0532d878846d3d17d6e0d833ac3cd9
parsl.dataflow.memoization INFO: Task 834 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279
parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/2bab4eee-6f6f-4c0d-9686-08d2bd86f5b5_isr_946074_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/2bab4eee-6f6f-4c0d-9686-08d2bd86f5b5_isr_946074_32.stderr
parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834
parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: <AppFuture at 0x152308802510 state=pending>
parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835
parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: <AppFuture at 0x152308802c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: <AppFuture at 0x152308803750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 837 has memoization hash c8089a528dbcd9521a90b3179d113635
parsl.dataflow.memoization INFO: Task 837 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280
parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2cae4652-eeb8-4124-adc5-83f6d46d0501_isr_946182_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2cae4652-eeb8-4124-adc5-83f6d46d0501_isr_946182_40.stderr
parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837
parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: <AppFuture at 0x152308803ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838
parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: <AppFuture at 0x15230880c090 state=pending>
parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: <AppFuture at 0x15230880cbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 342c6cd54b9a05cebb8674a9ee8156ad
parsl.dataflow.memoization INFO: Task 840 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088082c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281
parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1ae41411-f3ef-43cf-b42a-f75e658bc7b3_isr_946074_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1ae41411-f3ef-43cf-b42a-f75e658bc7b3_isr_946074_47.stderr
parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840
parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: <AppFuture at 0x15230880d410 state=pending>
parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841
parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: <AppFuture at 0x15230880db50 state=pending>
parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: <AppFuture at 0x15230880df90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 843 has memoization hash d14da21d50d7ae42485815f6fa7d6ee5
parsl.dataflow.memoization INFO: Task 843 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088085e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282
parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/65a573ed-83de-4c08-91db-8be7203525df_isr_946181_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/65a573ed-83de-4c08-91db-8be7203525df_isr_946181_27.stderr
parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843
parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: <AppFuture at 0x15230880e710 state=pending>
parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844
parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: <AppFuture at 0x15230880f010 state=pending>
parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: <AppFuture at 0x15230880f410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 846 has memoization hash c1618885260e3670a1272bcf999382c3
parsl.dataflow.memoization INFO: Task 846 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283
parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/0eb66183-6b81-44e2-906d-4b839253272c_isr_946072_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/0eb66183-6b81-44e2-906d-4b839253272c_isr_946072_33.stderr
parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846
parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: <AppFuture at 0x15230880f790 state=pending>
parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847
parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: <AppFuture at 0x15230880f950 state=pending>
parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: <AppFuture at 0x152308814790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 81cb9a44536e39874cbea80cd38e3034
parsl.dataflow.memoization INFO: Task 849 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284
parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/37a20b03-03c3-440c-85c5-544902099dd4_isr_946181_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/37a20b03-03c3-440c-85c5-544902099dd4_isr_946181_26.stderr
parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849
parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: <AppFuture at 0x1523088148d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850
parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: <AppFuture at 0x152308815610 state=pending>
parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: <AppFuture at 0x152308815b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 512ca059f5944559b061ded0fdcbe5d6
parsl.dataflow.memoization INFO: Task 852 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285
parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/8e9f7def-0d65-4328-8b41-877a8394ace8_isr_946181_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/8e9f7def-0d65-4328-8b41-877a8394ace8_isr_946181_42.stderr
parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852
parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: <AppFuture at 0x1523088158d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853
parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: <AppFuture at 0x152308816990 state=pending>
parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: <AppFuture at 0x152308816f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 751152bec16e31fd1abaf5c090e56f95
parsl.dataflow.memoization INFO: Task 855 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286
parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b16e8c5-4260-4521-9ca6-36dd4bdbd539_isr_946182_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b16e8c5-4260-4521-9ca6-36dd4bdbd539_isr_946182_26.stderr
parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855
parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: <AppFuture at 0x152308816c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856
parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: <AppFuture at 0x152308817490 state=pending>
parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: <AppFuture at 0x152308824450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 210115e7f1bfb3a95a9b0c1f32281b1f
parsl.dataflow.memoization INFO: Task 858 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287
parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e7d03dc1-c2ab-48fb-afbe-127efcff763a_isr_946074_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e7d03dc1-c2ab-48fb-afbe-127efcff763a_isr_946074_24.stderr
parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858
parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: <AppFuture at 0x152308815090 state=pending>
parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859
parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: <AppFuture at 0x152308824910 state=pending>
parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: <AppFuture at 0x152308825890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 861 has memoization hash a44f8dde81b2db89604e091ab0d59805
parsl.dataflow.memoization INFO: Task 861 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088098a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288
parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/09c617c6-9338-441d-b274-c42e6474ad64_isr_946181_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/09c617c6-9338-441d-b274-c42e6474ad64_isr_946181_23.stderr
parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861
parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: <AppFuture at 0x152308826710 state=pending>
parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862
parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: <AppFuture at 0x152308826750 state=pending>
parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: <AppFuture at 0x152308826d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 864 has memoization hash f1dca7955d16c11c9f4d51b2c170b531
parsl.dataflow.memoization INFO: Task 864 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289
parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e50c1b84-2749-4f97-a3df-971bccdf3964_isr_946181_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e50c1b84-2749-4f97-a3df-971bccdf3964_isr_946181_12.stderr
parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864
parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: <AppFuture at 0x1523088262d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865
parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: <AppFuture at 0x152308827b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: <AppFuture at 0x152308827e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 867 has memoization hash caccd52027c1c49fa4536f7e20514867
parsl.dataflow.memoization INFO: Task 867 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290
parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/69473f3b-8e13-4557-92ac-816a759e1388_isr_946182_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/69473f3b-8e13-4557-92ac-816a759e1388_isr_946182_53.stderr
parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867
parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: <AppFuture at 0x152308827c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868
parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: <AppFuture at 0x152308830510 state=pending>
parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: <AppFuture at 0x152308831490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 870 has memoization hash e4baf5b6f0e3784430f4e63b5675cdb3
parsl.dataflow.memoization INFO: Task 870 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291
parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/141310e1-8e56-4bee-9127-355dcb6806ef_isr_946182_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/141310e1-8e56-4bee-9127-355dcb6806ef_isr_946182_56.stderr
parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870
parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: <AppFuture at 0x152308830b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871
parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: <AppFuture at 0x152308832310 state=pending>
parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: <AppFuture at 0x152308832910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 632be50013ee99d84fea462ea0b02e5b
parsl.dataflow.memoization INFO: Task 873 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292
parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/823b260f-2599-475a-aafb-72f29b17237b_isr_946181_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/823b260f-2599-475a-aafb-72f29b17237b_isr_946181_22.stderr
parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873
parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: <AppFuture at 0x152308832650 state=pending>
parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874
parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: <AppFuture at 0x152308833990 state=pending>
parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: <AppFuture at 0x152308833d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 876 has memoization hash a2b87ef81e17f9db94f59c4eb6e9ac79
parsl.dataflow.memoization INFO: Task 876 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293
parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/bd927200-a795-4d43-8442-25591ffd2c25_isr_946073_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/bd927200-a795-4d43-8442-25591ffd2c25_isr_946073_31.stderr
parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876
parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: <AppFuture at 0x152308833110 state=pending>
parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877
parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: <AppFuture at 0x152308834bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: <AppFuture at 0x152308835150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 3e76eb410777fc2a34728f8709e70d88
parsl.dataflow.memoization INFO: Task 879 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294
parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fed15b00-d53f-478e-909e-60d5f6db31c0_isr_946180_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fed15b00-d53f-478e-909e-60d5f6db31c0_isr_946180_14.stderr
parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879
parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: <AppFuture at 0x1523088345d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880
parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: <AppFuture at 0x152308835c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: <AppFuture at 0x152308836650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 7acb6bba393cc4dc3af637db83b19fc1
parsl.dataflow.memoization INFO: Task 882 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295
parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f4b22f69-bed1-4515-b682-8566aeeb9d90_isr_946180_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f4b22f69-bed1-4515-b682-8566aeeb9d90_isr_946180_49.stderr
parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882
parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: <AppFuture at 0x152308836790 state=pending>
parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883
parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: <AppFuture at 0x152308837490 state=pending>
parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: <AppFuture at 0x152308837a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 6da41c8e810ac5f22dc464fd28c75bcb
parsl.dataflow.memoization INFO: Task 885 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296
parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/70640ead-f4d8-4914-83aa-560da32985ba_isr_946181_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/70640ead-f4d8-4914-83aa-560da32985ba_isr_946181_48.stderr
parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885
parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: <AppFuture at 0x152308837e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886
parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: <AppFuture at 0x1523088445d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: <AppFuture at 0x152308844dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 888 has memoization hash 992af664f45eb6b4eeebc6d742d0a208
parsl.dataflow.memoization INFO: Task 888 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297
parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/338e95fe-9473-4a21-a182-643c20e70561_isr_946182_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/338e95fe-9473-4a21-a182-643c20e70561_isr_946182_42.stderr
parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888
parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: <AppFuture at 0x15230d9c1050 state=pending>
parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889
parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: <AppFuture at 0x152308845c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: <AppFuture at 0x1523088461d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 891 has memoization hash dd18276aeafa4a87929f49fbd33a2d24
parsl.dataflow.memoization INFO: Task 891 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298
parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/521cc015-9db5-4074-a9b6-41268a09dac7_isr_946073_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/521cc015-9db5-4074-a9b6-41268a09dac7_isr_946073_52.stderr
parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891
parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: <AppFuture at 0x152308845f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892
parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: <AppFuture at 0x152308846c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: <AppFuture at 0x152308847590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 98726af768eca148ee683323ae3f2329
parsl.dataflow.memoization INFO: Task 894 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299
parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bc335be7-7cf0-4578-8824-85eb9af26292_isr_946072_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bc335be7-7cf0-4578-8824-85eb9af26292_isr_946072_31.stderr
parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894
parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: <AppFuture at 0x152308847890 state=pending>
parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895
parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: <AppFuture at 0x152308848210 state=pending>
parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: <AppFuture at 0x152308848a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 897 has memoization hash de6d512ef872c70164cdfe75e581c568
parsl.dataflow.memoization INFO: Task 897 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300
parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9f63037c-fb51-49f5-a754-260e748a8eb2_isr_946074_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9f63037c-fb51-49f5-a754-260e748a8eb2_isr_946074_58.stderr
parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897
parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: <AppFuture at 0x152308848b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898
parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: <AppFuture at 0x152308848290 state=pending>
parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: <AppFuture at 0x152308849dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 900 has memoization hash ebb10ee459864ce451542a7ed0d453af
parsl.dataflow.memoization INFO: Task 900 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301
parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fb3fe6c-6eb4-4953-a671-2f27340c1bcb_isr_946072_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fb3fe6c-6eb4-4953-a671-2f27340c1bcb_isr_946072_18.stderr
parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900
parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: <AppFuture at 0x152308951790 state=pending>
parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901
parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: <AppFuture at 0x15230884a610 state=pending>
parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: <AppFuture at 0x15230884b1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 903 has memoization hash a0c31679165e5a6dc33c1bb24d3f335d
parsl.dataflow.memoization INFO: Task 903 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302
parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ad8603f4-3ae2-497d-a5be-375b6a256a75_isr_946074_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ad8603f4-3ae2-497d-a5be-375b6a256a75_isr_946074_12.stderr
parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903
parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: <AppFuture at 0x15230884b310 state=pending>
parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904
parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: <AppFuture at 0x15230884b710 state=pending>
parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: <AppFuture at 0x1523088546d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 50e91d2ee4dc5b0bbcee4ae71d8a1fc9
parsl.dataflow.memoization INFO: Task 906 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303
parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/780b6721-9fa1-4bf6-bcd0-55511acd6ec0_isr_946182_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/780b6721-9fa1-4bf6-bcd0-55511acd6ec0_isr_946182_46.stderr
parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906
parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: <AppFuture at 0x152308854810 state=pending>
parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907
parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: <AppFuture at 0x152308855710 state=pending>
parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: <AppFuture at 0x152308855b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 909 has memoization hash c3086f9ccb6809614c3cdbb2a5f581e3
parsl.dataflow.memoization INFO: Task 909 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304
parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9b51743c-aea5-4e7f-9d5a-8fe50d32ca76_isr_946182_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9b51743c-aea5-4e7f-9d5a-8fe50d32ca76_isr_946182_49.stderr
parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909
parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: <AppFuture at 0x1523088557d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910
parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: <AppFuture at 0x152308856390 state=pending>
parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: <AppFuture at 0x152308856f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 912 has memoization hash c94026adf5cf286f35831d5b30d7b87e
parsl.dataflow.memoization INFO: Task 912 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305
parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80443b3f-f3f6-4ced-8ca9-ce020747df10_isr_946073_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80443b3f-f3f6-4ced-8ca9-ce020747df10_isr_946073_35.stderr
parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912
parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: <AppFuture at 0x15230a216510 state=pending>
parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913
parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: <AppFuture at 0x152308857fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: <AppFuture at 0x1523088683d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 915 has memoization hash d3ef31dcf5aa2308c47f28e02f62635b
parsl.dataflow.memoization INFO: Task 915 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306
parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c89dea4c-786e-4069-9012-2196fcba5326_isr_946181_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c89dea4c-786e-4069-9012-2196fcba5326_isr_946181_9.stderr
parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915
parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: <AppFuture at 0x152308868510 state=pending>
parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916
parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: <AppFuture at 0x152308869210 state=pending>
parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: <AppFuture at 0x152308869710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 918 has memoization hash e7b7a2959324f19bedc589c65de08cba
parsl.dataflow.memoization INFO: Task 918 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307
parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/688b831a-0a77-4777-8855-c384335162cf_isr_946074_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/688b831a-0a77-4777-8855-c384335162cf_isr_946074_19.stderr
parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918
parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: <AppFuture at 0x15231b250d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919
parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: <AppFuture at 0x15230886a610 state=pending>
parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: <AppFuture at 0x15230886ac50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 9e17265a2c19206d923f63a154b6bc3f
parsl.dataflow.memoization INFO: Task 921 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308
parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b5c02d06-0bda-4661-b639-9f79bac8b36b_isr_946181_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b5c02d06-0bda-4661-b639-9f79bac8b36b_isr_946181_62.stderr
parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921
parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: <AppFuture at 0x15230886a950 state=pending>
parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922
parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: <AppFuture at 0x15230886b550 state=pending>
parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: <AppFuture at 0x15230886bf50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 64ccfa54a29425147d61c99a6e589991
parsl.dataflow.memoization INFO: Task 924 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309
parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/41097f3e-372b-48d6-a088-cbf8e0e26fe2_isr_946073_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/41097f3e-372b-48d6-a088-cbf8e0e26fe2_isr_946073_43.stderr
parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924
parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: <AppFuture at 0x15230886bc90 state=pending>
parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925
parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: <AppFuture at 0x152308870850 state=pending>
parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: <AppFuture at 0x152308871390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 927 has memoization hash a2a7baaa6e1d9ee232363695d96fd382
parsl.dataflow.memoization INFO: Task 927 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310
parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9914b711-e456-44b4-82f3-4f57fd8c70dd_isr_946074_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9914b711-e456-44b4-82f3-4f57fd8c70dd_isr_946074_13.stderr
parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927
parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: <AppFuture at 0x152308871510 state=pending>
parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928
parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: <AppFuture at 0x152308871b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: <AppFuture at 0x152308872790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 930 has memoization hash f7748f6a7d9dbaaadc056a6a4d9bc4a8
parsl.dataflow.memoization INFO: Task 930 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311
parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6ac0a71a-2e07-47a2-b5e0-1289a8a0c97a_isr_946073_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6ac0a71a-2e07-47a2-b5e0-1289a8a0c97a_isr_946073_53.stderr
parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930
parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: <AppFuture at 0x1523088721d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931
parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: <AppFuture at 0x152308873590 state=pending>
parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: <AppFuture at 0x152308873b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 933 has memoization hash ff28a23f99a2a6a1412df5cf289d202e
parsl.dataflow.memoization INFO: Task 933 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312
parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3b4f7912-0c1c-42e3-852c-3ae99a24a784_isr_946182_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3b4f7912-0c1c-42e3-852c-3ae99a24a784_isr_946182_3.stderr
parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933
parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: <AppFuture at 0x152308873f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934
parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: <AppFuture at 0x1523088787d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: <AppFuture at 0x152308879010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 936 has memoization hash c58e68e8994734e8e492c82a9dc0279a
parsl.dataflow.memoization INFO: Task 936 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313
parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7c434cbb-efd9-4983-9c25-39539998ca79_isr_946181_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7c434cbb-efd9-4983-9c25-39539998ca79_isr_946181_44.stderr
parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936
parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: <AppFuture at 0x152308878610 state=pending>
parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937
parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: <AppFuture at 0x152308878290 state=pending>
parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: <AppFuture at 0x15230887a450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 34e2ee4625e13b1712bb3132902af04d
parsl.dataflow.memoization INFO: Task 939 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314
parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ddc4f1df-c947-41ea-bebe-bbbac5174f28_isr_946180_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ddc4f1df-c947-41ea-bebe-bbbac5174f28_isr_946180_23.stderr
parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939
parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: <AppFuture at 0x15230887b310 state=pending>
parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940
parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: <AppFuture at 0x15230887ac90 state=pending>
parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: <AppFuture at 0x15230887b810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 80d973a7380e0a1690222a6803bf0129
parsl.dataflow.memoization INFO: Task 942 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315
parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/66307685-3022-4947-a850-df1b47750461_isr_946074_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/66307685-3022-4947-a850-df1b47750461_isr_946074_42.stderr
parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942
parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: <AppFuture at 0x15230887bd50 state=pending>
parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943
parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: <AppFuture at 0x152308884550 state=pending>
parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: <AppFuture at 0x152308884c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 93a878d8437571c2b45c9cb584de7236
parsl.dataflow.memoization INFO: Task 945 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316
parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9e02ef19-218e-4e82-8a70-3218b3542a7e_isr_946074_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9e02ef19-218e-4e82-8a70-3218b3542a7e_isr_946074_28.stderr
parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945
parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: <AppFuture at 0x152308884790 state=pending>
parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946
parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: <AppFuture at 0x152308885cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: <AppFuture at 0x152308885d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 948 has memoization hash a30677552a1ebf51999e29b229f9a812
parsl.dataflow.memoization INFO: Task 948 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317
parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/4033d9bc-06be-45d6-a1a0-0b618a584ff5_isr_946181_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/4033d9bc-06be-45d6-a1a0-0b618a584ff5_isr_946181_24.stderr
parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948
parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: <AppFuture at 0x152308886710 state=pending>
parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949
parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: <AppFuture at 0x152308886e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: <AppFuture at 0x152308887450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 0af106549bb499de648150dbf725690c
parsl.dataflow.memoization INFO: Task 951 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318
parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1f88305c-8980-45c6-9ef0-8c37caeb987f_isr_946074_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1f88305c-8980-45c6-9ef0-8c37caeb987f_isr_946074_57.stderr
parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951
parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: <AppFuture at 0x152308887850 state=pending>
parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952
parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: <AppFuture at 0x152308856a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: <AppFuture at 0x15230888c890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 32a3e4c358f314282a26876450b6fdfa
parsl.dataflow.memoization INFO: Task 954 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319
parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5715258a-1d8d-4023-9345-7f11d066058c_isr_946181_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5715258a-1d8d-4023-9345-7f11d066058c_isr_946181_54.stderr
parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954
parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: <AppFuture at 0x152308885990 state=pending>
parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955
parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: <AppFuture at 0x15230888d550 state=pending>
parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: <AppFuture at 0x15230888dd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 957 has memoization hash ba645e44ba3171f6bdd3ff6b4d3fd387
parsl.dataflow.memoization INFO: Task 957 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320
parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c778f597-5baf-416c-9104-c2bb13cb532c_isr_946180_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c778f597-5baf-416c-9104-c2bb13cb532c_isr_946180_45.stderr
parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957
parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: <AppFuture at 0x15230888d710 state=pending>
parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958
parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: <AppFuture at 0x15230888e7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: <AppFuture at 0x15230888f210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 960 has memoization hash d69dcfd60063ee2ef5639ed63a0004fa
parsl.dataflow.memoization INFO: Task 960 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321
parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/106fa713-0f3a-4980-b159-ec04d8e302c3_isr_946074_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/106fa713-0f3a-4980-b159-ec04d8e302c3_isr_946074_29.stderr
parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960
parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: <AppFuture at 0x15230888ef10 state=pending>
parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961
parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: <AppFuture at 0x15230888f6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: <AppFuture at 0x1523088a0710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 70625a6c1e56404905050635c1922341
parsl.dataflow.memoization INFO: Task 963 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322
parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5b067585-4d55-49a3-a30d-e296e149c19d_isr_946074_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5b067585-4d55-49a3-a30d-e296e149c19d_isr_946074_33.stderr
parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963
parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: <AppFuture at 0x15230888f990 state=pending>
parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964
parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: <AppFuture at 0x1523088a0290 state=pending>
parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: <AppFuture at 0x1523088a1a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 68977d43ee69fe05e29414377aa510cc
parsl.dataflow.memoization INFO: Task 966 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323
parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/3598b994-7241-4a14-81a7-4fc36b258620_isr_946073_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/3598b994-7241-4a14-81a7-4fc36b258620_isr_946073_22.stderr
parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966
parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: <AppFuture at 0x1523088a1b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967
parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: <AppFuture at 0x1523088a2810 state=pending>
parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: <AppFuture at 0x1523088a2e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 503e92cc86d453d2bb43e44751a58fee
parsl.dataflow.memoization INFO: Task 969 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324
parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6bbc3558-56a0-41e9-a3be-b050662e5f88_isr_946072_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6bbc3558-56a0-41e9-a3be-b050662e5f88_isr_946072_37.stderr
parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969
parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: <AppFuture at 0x1523088a2f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970
parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: <AppFuture at 0x1523088a3850 state=pending>
parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: <AppFuture at 0x1523088b0090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 59c34364a6fb2e8cb75e773f927a51c9
parsl.dataflow.memoization INFO: Task 972 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325
parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/f2ac9769-c7ea-4024-b56a-ce5c2bb4cd91_isr_946072_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/f2ac9769-c7ea-4024-b56a-ce5c2bb4cd91_isr_946072_53.stderr
parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972
parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: <AppFuture at 0x1523088a3f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973
parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: <AppFuture at 0x1523088b1250 state=pending>
parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: <AppFuture at 0x1523088b1690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 975 has memoization hash d5db1f65d1e2bbba538a8301e45869c1
parsl.dataflow.memoization INFO: Task 975 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326
parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/12c28eee-2e3a-4bb6-8f66-21cadbbba13f_isr_946180_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/12c28eee-2e3a-4bb6-8f66-21cadbbba13f_isr_946180_58.stderr
parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975
parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: <AppFuture at 0x1523088b0a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976
parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: <AppFuture at 0x1523088b2110 state=pending>
parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: <AppFuture at 0x1523088b29d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 51483a58af9f91fc018b2d511a45008a
parsl.dataflow.memoization INFO: Task 978 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327
parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8e53e551-8330-4c43-8783-8546b12754bc_isr_946074_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8e53e551-8330-4c43-8783-8546b12754bc_isr_946074_43.stderr
parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978
parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: <AppFuture at 0x1523088b2350 state=pending>
parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979
parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: <AppFuture at 0x1523088b3250 state=pending>
parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: <AppFuture at 0x1523088b3dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 58e70d3edb843015f7b8b1eccdaeaa5c
parsl.dataflow.memoization INFO: Task 981 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328
parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/43e7c4c5-5e3e-4253-bd9e-cd52fb430452_isr_946073_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/43e7c4c5-5e3e-4253-bd9e-cd52fb430452_isr_946073_57.stderr
parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981
parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: <AppFuture at 0x152308954c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982
parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: <AppFuture at 0x1523088b4a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: <AppFuture at 0x1523088b5250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 984 has memoization hash fd87cb22220b4d8ae008bca51ca83390
parsl.dataflow.memoization INFO: Task 984 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329
parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ab5fb525-9d6c-40de-84d6-e3541a3bffe2_isr_946073_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ab5fb525-9d6c-40de-84d6-e3541a3bffe2_isr_946073_25.stderr
parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984
parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: <AppFuture at 0x1523088b4f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985
parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: <AppFuture at 0x1523088b6190 state=pending>
parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: <AppFuture at 0x1523088b6750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 6c846c655610d2bef8604319f9e4198b
parsl.dataflow.memoization INFO: Task 987 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330
parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ff0ed01a-71c4-4bcb-8fc0-f0fc929a9869_isr_946074_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ff0ed01a-71c4-4bcb-8fc0-f0fc929a9869_isr_946074_23.stderr
parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987
parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: <AppFuture at 0x1523088b5cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988
parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: <AppFuture at 0x1523088b7750 state=pending>
parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: <AppFuture at 0x1523088b7a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 78e5b79c02d2fbbd840102aa466f0230
parsl.dataflow.memoization INFO: Task 990 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331
parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/59f3c7b8-869e-4355-9643-7c4418b6bc4a_isr_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/59f3c7b8-869e-4355-9643-7c4418b6bc4a_isr_946072_61.stderr
parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990
parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: <AppFuture at 0x1523088b7d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991
parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: <AppFuture at 0x1523088c0150 state=pending>
parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: <AppFuture at 0x1523088c0f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 2e4259c1b1e6595e7977b8ca8ac9df35
parsl.dataflow.memoization INFO: Task 993 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332
parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47328e9f-cada-430d-b345-2443ff9ac690_isr_946073_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47328e9f-cada-430d-b345-2443ff9ac690_isr_946073_28.stderr
parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993
parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: <AppFuture at 0x1523088b1d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994
parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: <AppFuture at 0x1523088c0750 state=pending>
parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: <AppFuture at 0x1523088c2310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 996 has memoization hash 2cc9b2b2da4c2b67d9b4229a7bf35416
parsl.dataflow.memoization INFO: Task 996 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333
parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/74099723-cf51-4888-b38b-055685e8faae_isr_946182_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/74099723-cf51-4888-b38b-055685e8faae_isr_946182_16.stderr
parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996
parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: <AppFuture at 0x1523088c2950 state=pending>
parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997
parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: <AppFuture at 0x1523088c3210 state=pending>
parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: <AppFuture at 0x1523088c3790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 999 has memoization hash fa39d7421dcb246e606f78cdcf7721d6
parsl.dataflow.memoization INFO: Task 999 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334
parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8972a105-778c-4f14-b787-0c96941bfe4b_isr_946073_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8972a105-778c-4f14-b787-0c96941bfe4b_isr_946073_26.stderr
parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999
parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: <AppFuture at 0x1523088c34d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000
parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: <AppFuture at 0x1523088d0450 state=pending>
parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: <AppFuture at 0x1523088d0bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash ebc366254a414903ee238d4d0e025c9d
parsl.dataflow.memoization INFO: Task 1002 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335
parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/935bb7b5-ba14-4948-8150-b576851b8cac_isr_946074_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/935bb7b5-ba14-4948-8150-b576851b8cac_isr_946074_52.stderr
parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002
parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: <AppFuture at 0x1523088d0d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003
parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: <AppFuture at 0x1523088d1b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: <AppFuture at 0x1523088d1f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash fd9eaff85c2633d2b5374dade85a36e4
parsl.dataflow.memoization INFO: Task 1005 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336
parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9154de2b-9329-4eaa-a5b6-52f4936dd99e_isr_946074_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9154de2b-9329-4eaa-a5b6-52f4936dd99e_isr_946074_25.stderr
parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005
parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: <AppFuture at 0x1523088c1590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006
parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: <AppFuture at 0x1523088d2450 state=pending>
parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: <AppFuture at 0x1523088d32d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash d60a699b0145e04d68832ca9e847f4df
parsl.dataflow.memoization INFO: Task 1008 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337
parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/0bc7b2ab-2a01-418f-86e7-e42de1ed52a6_isr_946180_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/0bc7b2ab-2a01-418f-86e7-e42de1ed52a6_isr_946180_43.stderr
parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008
parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: <AppFuture at 0x1523088d3810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009
parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: <AppFuture at 0x1523088c2c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: <AppFuture at 0x1523086e46d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 8fe60a149bb61aa976f84d7de79c7051
parsl.dataflow.memoization INFO: Task 1011 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338
parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/12a418ab-7bd2-43a0-8ea9-75caa4adb229_isr_946181_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/12a418ab-7bd2-43a0-8ea9-75caa4adb229_isr_946181_7.stderr
parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011
parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: <AppFuture at 0x152308854d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012
parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: <AppFuture at 0x1523086e4bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: <AppFuture at 0x1523086e5c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash d523da09d2e809016589529bdf72a9d4
parsl.dataflow.memoization INFO: Task 1014 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339
parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7a24098d-09b8-4e7a-a273-0a125d151b40_isr_946182_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7a24098d-09b8-4e7a-a273-0a125d151b40_isr_946182_7.stderr
parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014
parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: <AppFuture at 0x1523086e6c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015
parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: <AppFuture at 0x1523086e6090 state=pending>
parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: <AppFuture at 0x1523086e7010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash aede78a8f9efa2f72a5438cca61dae03
parsl.dataflow.memoization INFO: Task 1017 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340
parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9bc224c4-b271-425b-8e6b-e31cc40e1447_isr_946073_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9bc224c4-b271-425b-8e6b-e31cc40e1447_isr_946073_44.stderr
parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017
parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: <AppFuture at 0x1523086e7190 state=pending>
parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018
parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: <AppFuture at 0x1523086e7f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: <AppFuture at 0x1523086ec410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 5bcb30d11119b2e6036c369dc7b02a4e
parsl.dataflow.memoization INFO: Task 1020 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341
parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9065cdfa-dfcb-445b-922c-1504ed3b144d_isr_946072_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9065cdfa-dfcb-445b-922c-1504ed3b144d_isr_946072_7.stderr
parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020
parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: <AppFuture at 0x1523086e7750 state=pending>
parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021
parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: <AppFuture at 0x1523086ece50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: <AppFuture at 0x1523086ed810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 9df5a8f701c642c5c386a471af4efe2e
parsl.dataflow.memoization INFO: Task 1023 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342
parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/28a8320e-0daa-4e44-8f5d-005d2aa48522_isr_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/28a8320e-0daa-4e44-8f5d-005d2aa48522_isr_946180_2.stderr
parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023
parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: <AppFuture at 0x1523086ed550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024
parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: <AppFuture at 0x1523086ee6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: <AppFuture at 0x1523086eec50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 390afc46801bbcde181a0c59becf9270
parsl.dataflow.memoization INFO: Task 1026 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343
parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/51f58ccf-1a4c-4553-b562-85da0a1724a2_isr_946182_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/51f58ccf-1a4c-4553-b562-85da0a1724a2_isr_946182_37.stderr
parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026
parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: <AppFuture at 0x1523086edf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027
parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: <AppFuture at 0x1523086ef9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: <AppFuture at 0x1523086effd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 6c98d74a35e78991043a8c8bb5c97bc5
parsl.dataflow.memoization INFO: Task 1029 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344
parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7c19adf3-844b-4715-80a3-72b221f59e9a_isr_946182_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7c19adf3-844b-4715-80a3-72b221f59e9a_isr_946182_55.stderr
parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029
parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: <AppFuture at 0x1523086efd10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030
parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: <AppFuture at 0x1523086f8890 state=pending>
parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: <AppFuture at 0x1523086f9390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 8b598d477588ec0a4bcd125d54126cc5
parsl.dataflow.memoization INFO: Task 1032 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345
parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/fb419264-3050-43fe-aca8-6ab5b8350755_isr_946181_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/fb419264-3050-43fe-aca8-6ab5b8350755_isr_946181_15.stderr
parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032
parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: <AppFuture at 0x152308870790 state=pending>
parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033
parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: <AppFuture at 0x1523086fa1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: <AppFuture at 0x1523086fa850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 7593d2dc93d1552323edfb93ccbe754c
parsl.dataflow.memoization INFO: Task 1035 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346
parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/28cf9bba-2443-48ca-b20a-c45c4b4fb92f_isr_946182_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/28cf9bba-2443-48ca-b20a-c45c4b4fb92f_isr_946182_29.stderr
parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035
parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: <AppFuture at 0x1523086fa590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036
parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: <AppFuture at 0x1523086fb6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: <AppFuture at 0x1523086fbbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 202423979ef6f93a2321c77b11b159c7
parsl.dataflow.memoization INFO: Task 1038 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347
parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/441f4339-7b68-4d53-8726-0b401b433ce2_isr_946181_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/441f4339-7b68-4d53-8726-0b401b433ce2_isr_946181_16.stderr
parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038
parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: <AppFuture at 0x1523086fb910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039
parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: <AppFuture at 0x1523086fc7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: <AppFuture at 0x1523086fd010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 86640b50e4f4c4c2f5160320f52a421c
parsl.dataflow.memoization INFO: Task 1041 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348
parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d6fdf13f-b70c-4704-8c4e-fd1101938f36_isr_946072_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d6fdf13f-b70c-4704-8c4e-fd1101938f36_isr_946072_20.stderr
parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041
parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: <AppFuture at 0x1523086fc550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042
parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: <AppFuture at 0x1523086fdf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: <AppFuture at 0x1523086fe450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 5468312458ffcff43289cd9a923956f9
parsl.dataflow.memoization INFO: Task 1044 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349
parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/71d6bfa2-00e7-428b-b3d3-fc1ed2996b1b_isr_946181_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/71d6bfa2-00e7-428b-b3d3-fc1ed2996b1b_isr_946181_40.stderr
parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044
parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: <AppFuture at 0x1523086fe590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045
parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: <AppFuture at 0x1523086feb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: <AppFuture at 0x1523086ff8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 0d7e88f85791cf5f814d1fe8255335d1
parsl.dataflow.memoization INFO: Task 1047 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350
parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/def43128-3d27-4710-bde4-f58e064b6ed8_isr_946181_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/def43128-3d27-4710-bde4-f58e064b6ed8_isr_946181_11.stderr
parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047
parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: <AppFuture at 0x1523086ffa10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048
parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: <AppFuture at 0x15230870c0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: <AppFuture at 0x15230870ccd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 8103fa90fafe3102942d199b33133fe9
parsl.dataflow.memoization INFO: Task 1050 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351
parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b4b8f4f6-d9a0-4b7f-a5d6-e821f8564b70_isr_946181_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b4b8f4f6-d9a0-4b7f-a5d6-e821f8564b70_isr_946181_4.stderr
parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050
parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: <AppFuture at 0x15230870c090 state=pending>
parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051
parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: <AppFuture at 0x15230870ddd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: <AppFuture at 0x15230870dd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 97111f522a5feb7d18e92b28d4758bb5
parsl.dataflow.memoization INFO: Task 1053 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352
parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb3943d6-6b36-4fb3-ac18-00303f67c1c8_isr_946182_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb3943d6-6b36-4fb3-ac18-00303f67c1c8_isr_946182_41.stderr
parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053
parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: <AppFuture at 0x15230870f110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054
parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: <AppFuture at 0x15230870f050 state=pending>
parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: <AppFuture at 0x15230870f510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 54979c5d0aaaaf91a8164e65d241d50b
parsl.dataflow.memoization INFO: Task 1056 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353
parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/78b8a866-1f10-4b93-849f-d5b4782882e7_isr_946182_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/78b8a866-1f10-4b93-849f-d5b4782882e7_isr_946182_38.stderr
parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056
parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: <AppFuture at 0x15230870fed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057
parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: <AppFuture at 0x15230870f9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: <AppFuture at 0x1523087189d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 3369fe81d6e7829b16ca54e643bbe3b3
parsl.dataflow.memoization INFO: Task 1059 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354
parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7f7368aa-4b91-465b-a72b-8afca85924a7_isr_946181_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7f7368aa-4b91-465b-a72b-8afca85924a7_isr_946181_17.stderr
parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059
parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: <AppFuture at 0x152308718710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060
parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: <AppFuture at 0x152308719910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: <AppFuture at 0x152308719e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 236d6889695f0c7da0060bf420478d49
parsl.dataflow.memoization INFO: Task 1062 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355
parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d69402cc-b3df-4d95-a5db-13a44f5ceadc_isr_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d69402cc-b3df-4d95-a5db-13a44f5ceadc_isr_946073_2.stderr
parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062
parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: <AppFuture at 0x152308719b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063
parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: <AppFuture at 0x15230871a310 state=pending>
parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: <AppFuture at 0x15230871b250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 87f987aac4c18d3f35c39703879b4525
parsl.dataflow.memoization INFO: Task 1065 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356
parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d6d9786f-78f2-43c0-9b34-6789ba7e4985_isr_946180_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d6d9786f-78f2-43c0-9b34-6789ba7e4985_isr_946180_57.stderr
parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065
parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: <AppFuture at 0x15230871b390 state=pending>
parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066
parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: <AppFuture at 0x15230871b710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: <AppFuture at 0x1523087246d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 81b54ef4dbba6963513287e25922aa02
parsl.dataflow.memoization INFO: Task 1068 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357
parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b054a092-be49-4975-8c11-49c03f8053e8_isr_946074_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b054a092-be49-4975-8c11-49c03f8053e8_isr_946074_62.stderr
parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068
parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: <AppFuture at 0x152308725510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069
parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: <AppFuture at 0x1523087250d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: <AppFuture at 0x152308725a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 2ea212991513fd6a9f33a2b0065ae2b1
parsl.dataflow.memoization INFO: Task 1071 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358
parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b90b0567-cc32-4a9b-b6ad-852a4d28f2b6_isr_946180_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b90b0567-cc32-4a9b-b6ad-852a4d28f2b6_isr_946180_35.stderr
parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071
parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: <AppFuture at 0x152308725bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072
parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: <AppFuture at 0x152308726310 state=pending>
parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: <AppFuture at 0x152308726e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 497e3ef5b8735e282ccb4b80321289a2
parsl.dataflow.memoization INFO: Task 1074 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359
parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce2d451d-4e54-4888-9d43-f8e3e5653b4d_isr_946180_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce2d451d-4e54-4888-9d43-f8e3e5653b4d_isr_946180_6.stderr
parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074
parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: <AppFuture at 0x152308727010 state=pending>
parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075
parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: <AppFuture at 0x152308727810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: <AppFuture at 0x1523087300d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 0c2a3612fe586b7ca8225ca1546a8d61
parsl.dataflow.memoization INFO: Task 1077 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360
parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fba59dfd-aaa5-4526-be5a-0e7874c91e52_isr_946074_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fba59dfd-aaa5-4526-be5a-0e7874c91e52_isr_946074_8.stderr
parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077
parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: <AppFuture at 0x152308727dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078
parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: <AppFuture at 0x152308730910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: <AppFuture at 0x152308731590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 0ebdcfa697c302b83a3b3b76905da0b4
parsl.dataflow.memoization INFO: Task 1080 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361
parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0379898b-d14b-422d-ae43-8f6777be9e58_isr_946182_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0379898b-d14b-422d-ae43-8f6777be9e58_isr_946182_14.stderr
parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080
parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: <AppFuture at 0x1523087326d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081
parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: <AppFuture at 0x152308732390 state=pending>
parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: <AppFuture at 0x152308732a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash a7b746b512bd5a49122fda5508ffb1e6
parsl.dataflow.memoization INFO: Task 1083 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362
parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3ce09ee6-7373-4a6d-8aa4-cabb54bee44b_isr_946072_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3ce09ee6-7373-4a6d-8aa4-cabb54bee44b_isr_946072_6.stderr
parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083
parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: <AppFuture at 0x1523087323d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084
parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: <AppFuture at 0x152308733290 state=pending>
parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: <AppFuture at 0x152308733dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash d4f9b80dbed443435a70ad477f4f1546
parsl.dataflow.memoization INFO: Task 1086 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363
parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/aaad56f7-bc85-4e2d-b21e-b1beb3e0ae34_isr_946073_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/aaad56f7-bc85-4e2d-b21e-b1beb3e0ae34_isr_946073_12.stderr
parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086
parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: <AppFuture at 0x152308733a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087
parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: <AppFuture at 0x15230873c610 state=pending>
parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: <AppFuture at 0x15230873d1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 7e6ec0e9b30f72fe919864f88495eb95
parsl.dataflow.memoization INFO: Task 1089 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364
parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5050735e-e323-4448-bf72-e317b301dd76_isr_946181_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5050735e-e323-4448-bf72-e317b301dd76_isr_946181_8.stderr
parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089
parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: <AppFuture at 0x15230873c750 state=pending>
parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090
parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: <AppFuture at 0x15230873e1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: <AppFuture at 0x15230873e690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 37f49ae6a45609a41152aad056691dc8
parsl.dataflow.memoization INFO: Task 1092 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365
parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c7d1c6b9-ba96-452d-92b3-20046ee31635_isr_946072_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c7d1c6b9-ba96-452d-92b3-20046ee31635_isr_946072_56.stderr
parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092
parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: <AppFuture at 0x15230873c850 state=pending>
parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093
parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: <AppFuture at 0x15230873f510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: <AppFuture at 0x15230873fa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash 7c9151a10615af2b425efcdf14f29ea2
parsl.dataflow.memoization INFO: Task 1095 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366
parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/717f4f1f-a0e3-4828-a4b2-8e0a5a49fe7b_isr_946182_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/717f4f1f-a0e3-4828-a4b2-8e0a5a49fe7b_isr_946182_27.stderr
parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095
parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: <AppFuture at 0x15230873fe10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096
parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: <AppFuture at 0x15230874c950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: <AppFuture at 0x15230874ce50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 6e1865cc1fdfac1ba88efc01aaeb1b2f
parsl.dataflow.memoization INFO: Task 1098 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367
parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e8b4c20-0116-4b44-a86c-d4b5cdd575b2_isr_946073_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e8b4c20-0116-4b44-a86c-d4b5cdd575b2_isr_946073_4.stderr
parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098
parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: <AppFuture at 0x15230874d290 state=pending>
parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099
parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: <AppFuture at 0x15230874d350 state=pending>
parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: <AppFuture at 0x15230874e310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 48cc9457c1d875ba96661f5d03467354
parsl.dataflow.memoization INFO: Task 1101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368
parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/62647372-819e-4b3c-b727-60171e1b0df0_isr_946180_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/62647372-819e-4b3c-b727-60171e1b0df0_isr_946180_42.stderr
parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101
parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: <AppFuture at 0x15230874e050 state=pending>
parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102
parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: <AppFuture at 0x15230874f390 state=pending>
parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: <AppFuture at 0x15230874f790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash c56345137bfba33522ea6a4b250f4f52
parsl.dataflow.memoization INFO: Task 1104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369
parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f454fbba-5ebb-4059-b0a0-64a4b0383ce4_isr_946182_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f454fbba-5ebb-4059-b0a0-64a4b0383ce4_isr_946182_5.stderr
parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104
parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: <AppFuture at 0x15230874f4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105
parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: <AppFuture at 0x1523087506d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: <AppFuture at 0x152308750b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash cc86d60efe8b22d6b39a06b378ccc00f
parsl.dataflow.memoization INFO: Task 1107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370
parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c183a6f7-f691-4811-819b-83e332d19f34_isr_946074_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c183a6f7-f691-4811-819b-83e332d19f34_isr_946074_55.stderr
parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107
parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: <AppFuture at 0x1523087508d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108
parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: <AppFuture at 0x152308751bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: <AppFuture at 0x152308751fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash b823a5839db002a125ae7f31b7729dc4
parsl.dataflow.memoization INFO: Task 1110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371
parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/80016d4d-d8ad-4b90-83c0-85551206b34c_isr_946074_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/80016d4d-d8ad-4b90-83c0-85551206b34c_isr_946074_36.stderr
parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110
parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: <AppFuture at 0x152308752950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111
parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: <AppFuture at 0x152308753090 state=pending>
parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: <AppFuture at 0x152308753450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 3fec2f5caeddf272c67ad5ae20f72be6
parsl.dataflow.memoization INFO: Task 1113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308736020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372
parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9ae62eca-7fe4-4a20-b9bb-605573883ed3_isr_946181_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9ae62eca-7fe4-4a20-b9bb-605573883ed3_isr_946181_6.stderr
parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113
parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: <AppFuture at 0x152308753910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114
parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: <AppFuture at 0x1523088d2810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping
2024-03-13 21:34:13 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/0 running/pending blocks, and 0 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 92 more blocks
parsl.executors.status_handling INFO: Scaling out by 92 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-13 21:34:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-13 21:34:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-13 21:34:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:14 proc_lsst.multi:170 [INFO]  [multi] job_id 17023666
proc_lsst.multi INFO: [multi] job_id 17023666
2024-03-13 21:34:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:14 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023666
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023666
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17023666
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-13 21:34:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-13 21:34:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:14 proc_lsst.multi:170 [INFO]  [multi] job_id 17023667
proc_lsst.multi INFO: [multi] job_id 17023667
2024-03-13 21:34:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:14 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023667
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023667
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17023667
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-13 21:34:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-13 21:34:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:15 proc_lsst.multi:170 [INFO]  [multi] job_id 17023668
proc_lsst.multi INFO: [multi] job_id 17023668
2024-03-13 21:34:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:15 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023668
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023668
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17023668
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-13 21:34:15 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-13 21:34:15 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:15 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:15 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:15 proc_lsst.multi:170 [INFO]  [multi] job_id 17023669
proc_lsst.multi INFO: [multi] job_id 17023669
2024-03-13 21:34:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:15 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023669
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023669
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17023669
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-13 21:34:15 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-13 21:34:15 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:15 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:15 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:15 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:15 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:16 proc_lsst.multi:170 [INFO]  [multi] job_id 17023670
proc_lsst.multi INFO: [multi] job_id 17023670
2024-03-13 21:34:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:16 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023670
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023670
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17023670
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-13 21:34:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-13 21:34:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:16 proc_lsst.multi:170 [INFO]  [multi] job_id 17023671
proc_lsst.multi INFO: [multi] job_id 17023671
2024-03-13 21:34:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-13 21:34:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:16 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023671
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023671
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17023671
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-13 21:34:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-13 21:34:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-13 21:34:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:17 proc_lsst.multi:170 [INFO]  [multi] job_id 17023672
proc_lsst.multi INFO: [multi] job_id 17023672
2024-03-13 21:34:17 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-13 21:34:17 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:17 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023672
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023672
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17023672
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-13 21:34:17 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-13 21:34:17 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:17 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:17 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:17 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:17 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-13 21:34:17 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:17 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:18 proc_lsst.multi:170 [INFO]  [multi] job_id 17023673
proc_lsst.multi INFO: [multi] job_id 17023673
2024-03-13 21:34:18 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-13 21:34:18 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:18 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023673
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023673
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17023673
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-13 21:34:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-13 21:34:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-13 21:34:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:19 proc_lsst.multi:170 [INFO]  [multi] job_id 17023674
proc_lsst.multi INFO: [multi] job_id 17023674
2024-03-13 21:34:19 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-13 21:34:19 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:19 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023674
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023674
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17023674
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-13 21:34:19 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-13 21:34:19 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:19 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-13 21:34:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:19 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:20 proc_lsst.multi:170 [INFO]  [multi] job_id 17023675
proc_lsst.multi INFO: [multi] job_id 17023675
2024-03-13 21:34:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-13 21:34:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:20 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023675
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023675
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17023675
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-13 21:34:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-13 21:34:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-13 21:34:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:21 proc_lsst.multi:170 [INFO]  [multi] job_id 17023676
proc_lsst.multi INFO: [multi] job_id 17023676
2024-03-13 21:34:21 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-13 21:34:21 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:21 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023676
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023676
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17023676
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-13 21:34:21 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-13 21:34:21 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:21 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-13 21:34:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:21 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:22 proc_lsst.multi:170 [INFO]  [multi] job_id 17023677
proc_lsst.multi INFO: [multi] job_id 17023677
2024-03-13 21:34:22 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-13 21:34:22 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:22 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023677
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023677
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17023677
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-13 21:34:22 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-13 21:34:22 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:22 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-13 21:34:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:22 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:23 proc_lsst.multi:170 [INFO]  [multi] job_id 17023678
proc_lsst.multi INFO: [multi] job_id 17023678
2024-03-13 21:34:23 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-13 21:34:23 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:23 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023678
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023678
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17023678
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-13 21:34:23 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-13 21:34:23 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:23 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-13 21:34:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:23 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:23 proc_lsst.multi:170 [INFO]  [multi] job_id 17023679
proc_lsst.multi INFO: [multi] job_id 17023679
2024-03-13 21:34:23 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-13 21:34:23 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:23 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023679
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023679
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17023679
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-13 21:34:23 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-13 21:34:23 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:23 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-13 21:34:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:23 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:24 proc_lsst.multi:170 [INFO]  [multi] job_id 17023680
proc_lsst.multi INFO: [multi] job_id 17023680
2024-03-13 21:34:24 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-13 21:34:24 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:24 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023680
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023680
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17023680
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-13 21:34:24 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-13 21:34:24 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:24 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-13 21:34:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:24 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:25 proc_lsst.multi:170 [INFO]  [multi] job_id 17023681
proc_lsst.multi INFO: [multi] job_id 17023681
2024-03-13 21:34:25 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-13 21:34:25 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:25 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023681
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023681
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17023681
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-13 21:34:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-13 21:34:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:25 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-13 21:34:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:26 proc_lsst.multi:170 [INFO]  [multi] job_id 17023682
proc_lsst.multi INFO: [multi] job_id 17023682
2024-03-13 21:34:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-13 21:34:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:26 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023682
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023682
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17023682
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-13 21:34:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-13 21:34:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:26 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-13 21:34:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17023683
proc_lsst.multi INFO: [multi] job_id 17023683
2024-03-13 21:34:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-13 21:34:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:28 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023683
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023683
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17023683
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-13 21:34:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-13 21:34:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-13 21:34:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17023684
proc_lsst.multi INFO: [multi] job_id 17023684
2024-03-13 21:34:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-13 21:34:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:29 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023684
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023684
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17023684
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-13 21:34:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-13 21:34:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-13 21:34:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17023685
proc_lsst.multi INFO: [multi] job_id 17023685
2024-03-13 21:34:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023685
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023685
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17023685
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-13 21:34:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-13 21:34:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17023686
proc_lsst.multi INFO: [multi] job_id 17023686
2024-03-13 21:34:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-13 21:34:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023686
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023686
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17023686
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-13 21:34:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-13 21:34:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-13 21:34:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17023687
proc_lsst.multi INFO: [multi] job_id 17023687
2024-03-13 21:34:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-13 21:34:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023687
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023687
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17023687
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-13 21:34:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-13 21:34:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-13 21:34:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17023688
proc_lsst.multi INFO: [multi] job_id 17023688
2024-03-13 21:34:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-13 21:34:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023688
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023688
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17023688
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-13 21:34:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-13 21:34:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-13 21:34:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17023689
proc_lsst.multi INFO: [multi] job_id 17023689
2024-03-13 21:34:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-13 21:34:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023689
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023689
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17023689
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-13 21:34:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-13 21:34:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-13 21:34:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:32 proc_lsst.multi:170 [INFO]  [multi] job_id 17023690
proc_lsst.multi INFO: [multi] job_id 17023690
2024-03-13 21:34:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-13 21:34:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023690
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023690
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17023690
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-13 21:34:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-13 21:34:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-13 21:34:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17023691
proc_lsst.multi INFO: [multi] job_id 17023691
2024-03-13 21:34:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-13 21:34:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023691
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023691
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17023691
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-13 21:34:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17023692
proc_lsst.multi INFO: [multi] job_id 17023692
2024-03-13 21:34:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-13 21:34:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023692
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023692
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17023692
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-13 21:34:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17023693
proc_lsst.multi INFO: [multi] job_id 17023693
2024-03-13 21:34:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-13 21:34:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023693
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023693
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17023693
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-13 21:34:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17023694
proc_lsst.multi INFO: [multi] job_id 17023694
2024-03-13 21:34:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-13 21:34:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023694
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023694
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17023694
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-13 21:34:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-13 21:34:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:34 proc_lsst.multi:170 [INFO]  [multi] job_id 17023695
proc_lsst.multi INFO: [multi] job_id 17023695
2024-03-13 21:34:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:34 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17023695
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17023695
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17023695
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-13 21:34:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:34 proc_lsst.multi:170 [INFO]  [multi] job_id 17023696
proc_lsst.multi INFO: [multi] job_id 17023696
2024-03-13 21:34:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:34 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17023696
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17023696
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17023696
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-13 21:34:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:34 proc_lsst.multi:170 [INFO]  [multi] job_id 17023697
proc_lsst.multi INFO: [multi] job_id 17023697
2024-03-13 21:34:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:34 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17023697
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17023697
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17023697
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-13 21:34:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:34 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cbe7b98f0f4dffe63c78d228196b2f6a with result from task 3
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 479ad88babad5a1704451be14a21f586
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 373
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecf736aa-2147-4f58-a2e6-702ed9a863c6_characterizeImage_946073_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecf736aa-2147-4f58-a2e6-702ed9a863c6_characterizeImage_946073_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80653471-3521-4223-a4b2-5ebfd10b8ec8_isr_946073_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80653471-3521-4223-a4b2-5ebfd10b8ec8_isr_946073_11.stderr
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4c49f5dcdb933fc05a86c31fc8b4b7c with result from task 0
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 4425cb2939407aa42e16611e603becfe
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 374
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dc3a5af1-fa70-4bfa-9773-400b0b779b49_characterizeImage_946074_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dc3a5af1-fa70-4bfa-9773-400b0b779b49_characterizeImage_946074_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/802a8304-af35-4868-87ea-56e4a557449d_isr_946074_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/802a8304-af35-4868-87ea-56e4a557449d_isr_946074_54.stderr
2024-03-13 21:34:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17023698
proc_lsst.multi INFO: [multi] job_id 17023698
2024-03-13 21:34:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17023698
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17023698
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17023698
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-13 21:34:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17023699
proc_lsst.multi INFO: [multi] job_id 17023699
2024-03-13 21:34:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17023699
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17023699
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17023699
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-13 21:34:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17023700
proc_lsst.multi INFO: [multi] job_id 17023700
2024-03-13 21:34:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17023700
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17023700
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17023700
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-13 21:34:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17023701
proc_lsst.multi INFO: [multi] job_id 17023701
2024-03-13 21:34:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023701
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023701
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17023701
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-13 21:34:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:35 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17023702
proc_lsst.multi INFO: [multi] job_id 17023702
2024-03-13 21:34:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023702
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023702
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17023702
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-13 21:34:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17023703
proc_lsst.multi INFO: [multi] job_id 17023703
2024-03-13 21:34:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023703
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023703
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17023703
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-13 21:34:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17023704
proc_lsst.multi INFO: [multi] job_id 17023704
2024-03-13 21:34:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023704
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023704
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17023704
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-13 21:34:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17023705
proc_lsst.multi INFO: [multi] job_id 17023705
2024-03-13 21:34:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023705
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023705
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17023705
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-13 21:34:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17023706
proc_lsst.multi INFO: [multi] job_id 17023706
2024-03-13 21:34:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-13 21:34:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023706
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023706
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17023706
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-13 21:34:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17023707
proc_lsst.multi INFO: [multi] job_id 17023707
2024-03-13 21:34:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-13 21:34:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023707
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023707
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17023707
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-13 21:34:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:36 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-13 21:34:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023708
proc_lsst.multi INFO: [multi] job_id 17023708
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023708
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023708
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17023708
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023709
proc_lsst.multi INFO: [multi] job_id 17023709
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023709
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023709
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17023709
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023710
proc_lsst.multi INFO: [multi] job_id 17023710
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023710
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023710
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17023710
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023711
proc_lsst.multi INFO: [multi] job_id 17023711
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023711
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023711
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17023711
parsl.executors.status_handling INFO: Allocated block ID 47
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023712
proc_lsst.multi INFO: [multi] job_id 17023712
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023712
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023712
parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17023712
parsl.executors.status_handling INFO: Allocated block ID 48
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023713
proc_lsst.multi INFO: [multi] job_id 17023713
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023713
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023713
parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17023713
parsl.executors.status_handling INFO: Allocated block ID 49
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023715
proc_lsst.multi INFO: [multi] job_id 17023715
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023715
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023715
parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17023715
parsl.executors.status_handling INFO: Allocated block ID 50
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17023716
proc_lsst.multi INFO: [multi] job_id 17023716
2024-03-13 21:34:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-13 21:34:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023716
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023716
parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17023716
parsl.executors.status_handling INFO: Allocated block ID 51
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51
2024-03-13 21:34:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-13 21:34:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17023717
proc_lsst.multi INFO: [multi] job_id 17023717
2024-03-13 21:34:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-13 21:34:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023717
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023717
parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17023717
parsl.executors.status_handling INFO: Allocated block ID 52
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52
2024-03-13 21:34:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17023718
proc_lsst.multi INFO: [multi] job_id 17023718
2024-03-13 21:34:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-13 21:34:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023718
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023718
parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17023718
parsl.executors.status_handling INFO: Allocated block ID 53
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53
2024-03-13 21:34:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17023719
proc_lsst.multi INFO: [multi] job_id 17023719
2024-03-13 21:34:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-13 21:34:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023719
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023719
parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17023719
parsl.executors.status_handling INFO: Allocated block ID 54
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54
2024-03-13 21:34:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17023720
proc_lsst.multi INFO: [multi] job_id 17023720
2024-03-13 21:34:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-13 21:34:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023720
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023720
parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17023720
parsl.executors.status_handling INFO: Allocated block ID 55
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55
2024-03-13 21:34:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17023721
proc_lsst.multi INFO: [multi] job_id 17023721
2024-03-13 21:34:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17023721
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17023721
parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17023721
parsl.executors.status_handling INFO: Allocated block ID 56
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56
2024-03-13 21:34:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:38 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-13 21:34:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17023722
proc_lsst.multi INFO: [multi] job_id 17023722
2024-03-13 21:34:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023722
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023722
parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17023722
parsl.executors.status_handling INFO: Allocated block ID 57
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57
2024-03-13 21:34:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17023723
proc_lsst.multi INFO: [multi] job_id 17023723
2024-03-13 21:34:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023723
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023723
parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17023723
parsl.executors.status_handling INFO: Allocated block ID 58
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58
2024-03-13 21:34:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17023724
proc_lsst.multi INFO: [multi] job_id 17023724
2024-03-13 21:34:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023724
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023724
parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17023724
parsl.executors.status_handling INFO: Allocated block ID 59
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59
2024-03-13 21:34:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17023725
proc_lsst.multi INFO: [multi] job_id 17023725
2024-03-13 21:34:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023725
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023725
parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17023725
parsl.executors.status_handling INFO: Allocated block ID 60
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60
2024-03-13 21:34:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:39 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-13 21:34:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17023726
proc_lsst.multi INFO: [multi] job_id 17023726
2024-03-13 21:34:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023726
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023726
parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17023726
parsl.executors.status_handling INFO: Allocated block ID 61
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61
2024-03-13 21:34:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17023727
proc_lsst.multi INFO: [multi] job_id 17023727
2024-03-13 21:34:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-13 21:34:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023727
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023727
parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17023727
parsl.executors.status_handling INFO: Allocated block ID 62
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62
2024-03-13 21:34:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17023728
proc_lsst.multi INFO: [multi] job_id 17023728
2024-03-13 21:34:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-13 21:34:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023728
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023728
parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17023728
parsl.executors.status_handling INFO: Allocated block ID 63
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63
2024-03-13 21:34:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17023729
proc_lsst.multi INFO: [multi] job_id 17023729
2024-03-13 21:34:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-13 21:34:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023729
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023729
parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17023729
parsl.executors.status_handling INFO: Allocated block ID 64
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64
2024-03-13 21:34:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17023730
proc_lsst.multi INFO: [multi] job_id 17023730
2024-03-13 21:34:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-13 21:34:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023730
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023730
parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17023730
parsl.executors.status_handling INFO: Allocated block ID 65
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65
2024-03-13 21:34:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17023731
proc_lsst.multi INFO: [multi] job_id 17023731
2024-03-13 21:34:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-13 21:34:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023731
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023731
parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17023731
parsl.executors.status_handling INFO: Allocated block ID 66
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66
2024-03-13 21:34:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:40 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-13 21:34:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023732
proc_lsst.multi INFO: [multi] job_id 17023732
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023732
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023732
parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17023732
parsl.executors.status_handling INFO: Allocated block ID 67
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023733
proc_lsst.multi INFO: [multi] job_id 17023733
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023733
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023733
parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17023733
parsl.executors.status_handling INFO: Allocated block ID 68
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023734
proc_lsst.multi INFO: [multi] job_id 17023734
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023734
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023734
parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17023734
parsl.executors.status_handling INFO: Allocated block ID 69
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023735
proc_lsst.multi INFO: [multi] job_id 17023735
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023735
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023735
parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17023735
parsl.executors.status_handling INFO: Allocated block ID 70
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023736
proc_lsst.multi INFO: [multi] job_id 17023736
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023736
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023736
parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17023736
parsl.executors.status_handling INFO: Allocated block ID 71
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023737
proc_lsst.multi INFO: [multi] job_id 17023737
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023737
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023737
parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17023737
parsl.executors.status_handling INFO: Allocated block ID 72
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023738
proc_lsst.multi INFO: [multi] job_id 17023738
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023738
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023738
parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17023738
parsl.executors.status_handling INFO: Allocated block ID 73
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17023739
proc_lsst.multi INFO: [multi] job_id 17023739
2024-03-13 21:34:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-13 21:34:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023739
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023739
parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17023739
parsl.executors.status_handling INFO: Allocated block ID 74
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74
2024-03-13 21:34:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:41 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-13 21:34:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023740
proc_lsst.multi INFO: [multi] job_id 17023740
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023740
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023740
parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17023740
parsl.executors.status_handling INFO: Allocated block ID 75
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023741
proc_lsst.multi INFO: [multi] job_id 17023741
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023741
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023741
parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17023741
parsl.executors.status_handling INFO: Allocated block ID 76
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023742
proc_lsst.multi INFO: [multi] job_id 17023742
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023742
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023742
parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17023742
parsl.executors.status_handling INFO: Allocated block ID 77
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023743
proc_lsst.multi INFO: [multi] job_id 17023743
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023743
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023743
parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17023743
parsl.executors.status_handling INFO: Allocated block ID 78
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023744
proc_lsst.multi INFO: [multi] job_id 17023744
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023744
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023744
parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17023744
parsl.executors.status_handling INFO: Allocated block ID 79
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023745
proc_lsst.multi INFO: [multi] job_id 17023745
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023745
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023745
parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17023745
parsl.executors.status_handling INFO: Allocated block ID 80
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023746
proc_lsst.multi INFO: [multi] job_id 17023746
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023746
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023746
parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17023746
parsl.executors.status_handling INFO: Allocated block ID 81
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023747
proc_lsst.multi INFO: [multi] job_id 17023747
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023747
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023747
parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17023747
parsl.executors.status_handling INFO: Allocated block ID 82
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17023748
proc_lsst.multi INFO: [multi] job_id 17023748
2024-03-13 21:34:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-13 21:34:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023748
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023748
parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17023748
parsl.executors.status_handling INFO: Allocated block ID 83
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83
2024-03-13 21:34:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:42 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-13 21:34:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023749
proc_lsst.multi INFO: [multi] job_id 17023749
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023749
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023749
parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17023749
parsl.executors.status_handling INFO: Allocated block ID 84
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023750
proc_lsst.multi INFO: [multi] job_id 17023750
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023750
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023750
parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17023750
parsl.executors.status_handling INFO: Allocated block ID 85
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023751
proc_lsst.multi INFO: [multi] job_id 17023751
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023751
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023751
parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17023751
parsl.executors.status_handling INFO: Allocated block ID 86
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023752
proc_lsst.multi INFO: [multi] job_id 17023752
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023752
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023752
parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17023752
parsl.executors.status_handling INFO: Allocated block ID 87
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023753
proc_lsst.multi INFO: [multi] job_id 17023753
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023753
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023753
parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17023753
parsl.executors.status_handling INFO: Allocated block ID 88
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023754
proc_lsst.multi INFO: [multi] job_id 17023754
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023754
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023754
parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17023754
parsl.executors.status_handling INFO: Allocated block ID 89
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023755
proc_lsst.multi INFO: [multi] job_id 17023755
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023755
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023755
parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17023755
parsl.executors.status_handling INFO: Allocated block ID 90
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17023756
proc_lsst.multi INFO: [multi] job_id 17023756
2024-03-13 21:34:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-13 21:34:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023756
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023756
parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17023756
parsl.executors.status_handling INFO: Allocated block ID 91
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91
2024-03-13 21:34:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:43 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-13 21:34:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17023757
proc_lsst.multi INFO: [multi] job_id 17023757
2024-03-13 21:34:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-13 21:34:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:44 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023757
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023757
parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17023757
parsl.executors.status_handling INFO: Allocated block ID 92
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92
2024-03-13 21:34:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
2024-03-13 21:34:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-13 21:34:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-13 21:34:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-13 21:34:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-13 21:34:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-13 21:34:44 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-13 21:34:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-13 21:34:44 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-13 21:34:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-13 21:34:44 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-13 21:34:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-13 21:34:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.9,n3009,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54727 --result_port=54577 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-13 21:34:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17023758
proc_lsst.multi INFO: [multi] job_id 17023758
2024-03-13 21:34:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-13 21:34:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-13 21:34:44 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17023758
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17023758
parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17023758
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:44 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:45 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:45 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:45 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:45 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:45 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:45 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:46 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:46 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:46 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:46 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:47 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:47 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:47 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:48 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:48 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:49 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:49 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:49 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:50 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:50 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4093ecf14fb1046829531897fc143a08 with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 7081284b8d14d7ac80a37d239a5d3fcf
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 375
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5f99f95e-cbf8-4ed9-843d-276b5fc7767e_characterizeImage_946072_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5f99f95e-cbf8-4ed9-843d-276b5fc7767e_characterizeImage_946072_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/98370b0f-235d-4a24-878a-50b488624732_isr_946072_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/98370b0f-235d-4a24-878a-50b488624732_isr_946072_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:51 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:51 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:51 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:51 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:52 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:52 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:53 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:53 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78435dfef5d35e3dbb11b4605ee64759 with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 6cb0c4e0c29e5583b2aa87fd98ee8e16
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 376
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/4c9174ac-14c6-4f5a-8770-7c7433c1c926_characterizeImage_946182_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/4c9174ac-14c6-4f5a-8770-7c7433c1c926_characterizeImage_946182_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0b8c9127-68e7-443f-ab6b-87c5b1392e7b_isr_946182_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0b8c9127-68e7-443f-ab6b-87c5b1392e7b_isr_946182_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:54 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:54 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:54 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:55 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:55 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:55 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:56 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:57 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:58 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:58 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:58 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:58 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:58 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:34:59 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:00 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:00 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:00 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:00 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:00 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:00 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:01 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:01 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:01 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:01 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:01 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:02 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:02 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:02 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:02 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:02 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:02 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 PD
17023751 PD
17023752 PD
17023753 PD
17023754 PD
17023755 PD
17023756 PD
17023757 PD
17023758 PD
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:03 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:03 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:04 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:04 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28c671455e2a8919d28ec606e6de2c0a with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash b47a276511ef58cd516d36f00e907304
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 377
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/17638c39-c928-4320-98bc-9eb0cb7c3a50_characterizeImage_946074_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/17638c39-c928-4320-98bc-9eb0cb7c3a50_characterizeImage_946074_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5a144e4b-acb4-4cb8-a444-4d624385db32_isr_946074_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5a144e4b-acb4-4cb8-a444-4d624385db32_isr_946074_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:04 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:04 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8a767ec427955db53d4b6cee4ba7ff3 with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 2368c6855749594972a03fdc4c8268dc
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 378
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7879cc40-cc26-416e-8d07-9cfdc544e561_characterizeImage_946073_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7879cc40-cc26-416e-8d07-9cfdc544e561_characterizeImage_946073_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/145714e5-1525-4254-be8a-e403bc385e6b_isr_946073_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/145714e5-1525-4254-be8a-e403bc385e6b_isr_946073_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:04 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f680fa1d39aa4ed2a8b0e957a71a2ca0 with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 53ef5333dc70d4429b905e01603177a4
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 379
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/97de818e-7075-4d8f-94a7-bc08342edb91_characterizeImage_946181_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/97de818e-7075-4d8f-94a7-bc08342edb91_characterizeImage_946181_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2c004503-eb71-40d1-805b-764e737a6a89_isr_946181_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2c004503-eb71-40d1-805b-764e737a6a89_isr_946181_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 91/2 running/pending blocks, and 265 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 91/2 running/pending blocks, and 265 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4670e632229d121546828e055471933 with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash a4cfbd906875fd4e8fbbc33a5126e031
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 380
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0d0973e1-c8ab-4062-b0a9-282092513c81_characterizeImage_946072_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0d0973e1-c8ab-4062-b0a9-282092513c81_characterizeImage_946072_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/214dcb40-cf1e-44f4-9643-0dd74ce7c983_isr_946072_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/214dcb40-cf1e-44f4-9643-0dd74ce7c983_isr_946072_40.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf434908d966c0d72d5383bd41f4d8bd with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 5ec32a15c8dc28c73e4436cf27f4b5e4
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 381
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/48e3e8e6-2cff-4ad3-9e85-48a6edb9d72e_characterizeImage_946180_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/48e3e8e6-2cff-4ad3-9e85-48a6edb9d72e_characterizeImage_946180_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c15f73bc-ce98-4b5a-a369-9240d8481637_isr_946180_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c15f73bc-ce98-4b5a-a369-9240d8481637_isr_946180_19.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da7e39fa7b8d7c3f2b98d4d178381f53 with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 4867fab6974ffa7507e65d4e20e67d8e
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 382
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/4bf35adf-74ad-4eba-9291-862380eca317_characterizeImage_946074_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/4bf35adf-74ad-4eba-9291-862380eca317_characterizeImage_946074_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/61f3b29d-be37-4ba3-a3da-e31c76320bdc_isr_946074_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/61f3b29d-be37-4ba3-a3da-e31c76320bdc_isr_946074_4.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70d9b4752836876b5d8674546816e0cd with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 78518e85415479c553d00c885eeb22b1
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 383
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/f73b3b4e-83f7-417b-99cc-2dafd01bb074_characterizeImage_946181_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/f73b3b4e-83f7-417b-99cc-2dafd01bb074_characterizeImage_946181_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7dc1f173-7bf4-4acb-b239-03ba3e113a97_isr_946181_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7dc1f173-7bf4-4acb-b239-03ba3e113a97_isr_946181_61.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a58305330994aac65f04dbafec20a16b with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 3524b97997fee1254c0b82d05af434c2
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 384
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a0e09ed3-555d-4572-aa0b-01f9edee5b6c_characterizeImage_946182_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a0e09ed3-555d-4572-aa0b-01f9edee5b6c_characterizeImage_946182_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3de6f9e0-8be0-47cf-a9b3-1fff75ee1295_isr_946182_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3de6f9e0-8be0-47cf-a9b3-1fff75ee1295_isr_946182_60.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90aea989930ebb9f700931da3b3fd777 with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash e8737ee17b1dabdaee0935ce78d31e9b
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 385
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9e39a269-89af-42d9-82b6-df35c55193ba_characterizeImage_946073_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9e39a269-89af-42d9-82b6-df35c55193ba_characterizeImage_946073_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d11b15f3-4367-463d-832a-10f721b1270f_isr_946073_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d11b15f3-4367-463d-832a-10f721b1270f_isr_946073_54.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 91/2 running/pending blocks, and 281 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3765022f60a11db9bd350136cb5768e9 with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 37a11fa9a733935adc5619fb8b5ae5f9
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 386
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/47cb0f5a-d920-4722-85ed-bb545b5de958_characterizeImage_946072_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/47cb0f5a-d920-4722-85ed-bb545b5de958_characterizeImage_946072_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e36c594a-b9bc-4cd7-a3c1-59e424792a69_isr_946072_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e36c594a-b9bc-4cd7-a3c1-59e424792a69_isr_946072_45.stderr
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:15 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:16 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:17 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:18 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:19 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0aa172e1253163bab226a9468d5cfff9 with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 156bf146e23665379759e040d5feacf0
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 387
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/80236381-566b-4146-930a-5d657323e390_characterizeImage_946182_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/80236381-566b-4146-930a-5d657323e390_characterizeImage_946182_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3a0e9159-e3ed-4b6d-92f2-437adc03151b_isr_946182_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3a0e9159-e3ed-4b6d-92f2-437adc03151b_isr_946182_28.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a950f1b358620eb7211df11d5f4d32d6 with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 6fdda165a83c37b2bd007fe42cc6021a
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 388
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/eed47b7c-4ca0-4f99-b7ff-1e836b6e2192_characterizeImage_946182_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/eed47b7c-4ca0-4f99-b7ff-1e836b6e2192_characterizeImage_946182_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/5544f9d8-7ee6-4cd5-9d06-d989eaa0c1e1_isr_946182_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/5544f9d8-7ee6-4cd5-9d06-d989eaa0c1e1_isr_946182_21.stderr
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e821c9614100efdc1356a93f81ba969 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 13c8b737d08675ab2f6c7cea731ccdd0
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 389
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/59593040-90e1-4e7e-9d58-ce6b1a7ccc44_characterizeImage_946181_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/59593040-90e1-4e7e-9d58-ce6b1a7ccc44_characterizeImage_946181_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3ea720bd-9f87-4cf5-a411-7ee95de0d8da_isr_946181_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3ea720bd-9f87-4cf5-a411-7ee95de0d8da_isr_946181_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:20 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:21 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:21 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:21 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:21 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:21 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:21 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 256fbaee1104d1dcbbbfd25ae7fea0e8 with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 8cc7101394c0c5e00fcb1e77ec213371
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 390
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8b6f98c0-1ce0-4415-882a-c56ddd6dde4b_characterizeImage_946074_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8b6f98c0-1ce0-4415-882a-c56ddd6dde4b_characterizeImage_946074_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/af4a3b76-43ed-4341-8231-92bda52dd70c_isr_946074_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/af4a3b76-43ed-4341-8231-92bda52dd70c_isr_946074_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:22 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:23 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:23 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a3ace04ee8bfe91dbc239982855cfd2 with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash b46dec850117b984aac14bc81dd4af28
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 391
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3794f6fa-cc0a-48cc-96bd-95eddfd843c1_characterizeImage_946180_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3794f6fa-cc0a-48cc-96bd-95eddfd843c1_characterizeImage_946180_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/21c01109-d822-4f8f-9c15-e836f31d9cd7_isr_946180_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/21c01109-d822-4f8f-9c15-e836f31d9cd7_isr_946180_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:23 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:24 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 725e7bd093833b70f0c516efa8aad1ba with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash b5a52d5f8cb0c7ffceed2f16cdbb33b8
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 392
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50ef5a04-6dab-4c5c-9085-b0e09b0aef6a_characterizeImage_946181_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50ef5a04-6dab-4c5c-9085-b0e09b0aef6a_characterizeImage_946181_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6739dd26-5da9-4aa9-b56a-51e08e261b3b_isr_946181_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6739dd26-5da9-4aa9-b56a-51e08e261b3b_isr_946181_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:24 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6f03d1075bbeda6b526631b230cf677 with result from task 153
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 19e9c259b7df45f83caa30592ba44ed8
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 393
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/751704a5-db7f-4f24-90e9-62eefa745427_characterizeImage_946072_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/751704a5-db7f-4f24-90e9-62eefa745427_characterizeImage_946072_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3d865830-82d8-4c48-a5c7-4cf5a468eefb_isr_946072_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3d865830-82d8-4c48-a5c7-4cf5a468eefb_isr_946072_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:25 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99f7ab28a31f55e90c9991585bfbd8f7 with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash a71e7052af1ad28ad78a50523c425810
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 394
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ec9fb21e-d89e-4498-9da3-42855dfd2f2f_characterizeImage_946180_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ec9fb21e-d89e-4498-9da3-42855dfd2f2f_characterizeImage_946180_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce3b1349-906c-4743-af52-2c80f3b9af03_isr_946180_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce3b1349-906c-4743-af52-2c80f3b9af03_isr_946180_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:25 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:25 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d1672d19872300e904d8f078917941e with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 53f29d3e714443e699036c87c1ccf242
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 395
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/59a768e0-5ec4-4776-ac99-be9d661e8073_characterizeImage_946180_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/59a768e0-5ec4-4776-ac99-be9d661e8073_characterizeImage_946180_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ef37a707-fb2a-4d71-9eea-6f5ee703ccdf_isr_946180_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ef37a707-fb2a-4d71-9eea-6f5ee703ccdf_isr_946180_61.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 185ca1dff4b0ffae291b748add9e4bf0 with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash c959292310298fc2641538df573712a4
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 396
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/74b3c647-b0fb-4323-afa2-933c658752a0_characterizeImage_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/74b3c647-b0fb-4323-afa2-933c658752a0_characterizeImage_946074_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/6fde5dc1-9890-4d9d-90ad-a2095b5fe440_isr_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/6fde5dc1-9890-4d9d-90ad-a2095b5fe440_isr_946074_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:25 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:25 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:26 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:26 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:26 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:26 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:26 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:26 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85114afb7f590dadc2f1bd5dd8a42ea1 with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 6c9ac69d7d2a8c6deca540eaa3a9a682
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 397
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/38079b15-5366-4b15-815e-6e03ba53ee61_characterizeImage_946072_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/38079b15-5366-4b15-815e-6e03ba53ee61_characterizeImage_946072_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9a3d01ff-13db-47bd-b97f-a8899027db36_isr_946072_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9a3d01ff-13db-47bd-b97f-a8899027db36_isr_946072_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a048eb96d666190d725dd2543841b9d with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash e5dee24e21d5685cdf661fe54495c7e6
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 398
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7cb957a4-40ad-4286-8b4f-3976f1c98080_characterizeImage_946073_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7cb957a4-40ad-4286-8b4f-3976f1c98080_characterizeImage_946073_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e5819ef-e110-441a-8a5c-b4a23756d079_isr_946073_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e5819ef-e110-441a-8a5c-b4a23756d079_isr_946073_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:27 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:28 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:28 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f192ac424acacdcb302246ad476b0aca with result from task 186
parsl.dataflow.dflow DEBUG: Task 186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash fe582d2dc4b43e40d1bcfecefbda0a8e
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 399
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b4c732b-5048-4abf-bc39-1ae4c0165aff_characterizeImage_946073_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b4c732b-5048-4abf-bc39-1ae4c0165aff_characterizeImage_946073_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4f964645-ba71-4069-ae5c-75ca4a71de3c_isr_946073_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4f964645-ba71-4069-ae5c-75ca4a71de3c_isr_946073_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:28 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:28 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:28 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:29 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 623ae8612047447831b7cf9572f6808e with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash f8ac2f50331004d3d2fa2725e1b24e2a
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fccb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 400
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a6c92cb1-d95e-4539-9b7c-fed22924c4e0_characterizeImage_946073_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a6c92cb1-d95e-4539-9b7c-fed22924c4e0_characterizeImage_946073_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/21dda898-c035-4543-abdb-339d0caa6750_isr_946073_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/21dda898-c035-4543-abdb-339d0caa6750_isr_946073_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:29 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 248e68c22c23e4e5b3ee17ef07065803 with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash dc9781abfa6a60a4f4b73b7c505bda60
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 401
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a0bccbb6-5e16-476d-8649-b85ad57f9e9a_characterizeImage_946180_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a0bccbb6-5e16-476d-8649-b85ad57f9e9a_characterizeImage_946180_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f2a3047e-a0ed-4bd3-8694-297ff2c6428b_isr_946180_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f2a3047e-a0ed-4bd3-8694-297ff2c6428b_isr_946180_16.stderr
parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8bebe3b98e248abef78f244662639d2a with result from task 426
parsl.dataflow.dflow DEBUG: Task 426 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 427 has memoization hash cc4c9ba71f5927cae5d5779b23eabf97
parsl.dataflow.memoization INFO: Task 427 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 402
parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/692e0adf-8a7f-4bc7-a454-724d58ca2a15_characterizeImage_946074_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/692e0adf-8a7f-4bc7-a454-724d58ca2a15_characterizeImage_946074_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/170a23e3-2c1d-41af-9918-b197dea035aa_isr_946074_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/170a23e3-2c1d-41af-9918-b197dea035aa_isr_946074_45.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5eb72e8574a68077db490fad06eaf4ac with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash a86c60c535148267711606d5c02a83c1
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 403
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/35384805-789c-45d7-8505-1dc6a1490223_characterizeImage_946072_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/35384805-789c-45d7-8505-1dc6a1490223_characterizeImage_946072_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6206ed07-77cf-46de-a47f-6441a3ca4a5c_isr_946072_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6206ed07-77cf-46de-a47f-6441a3ca4a5c_isr_946072_10.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f202a016376af973e55cc8a5ff655447 with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 6749c2160246c5bc4bb1bca45e48d88c
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 404
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6e4e577b-e260-455c-aef3-0da6291a6c41_characterizeImage_946180_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6e4e577b-e260-455c-aef3-0da6291a6c41_characterizeImage_946180_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ec384bf3-a8d8-475b-957e-05ffe0b19e83_isr_946180_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ec384bf3-a8d8-475b-957e-05ffe0b19e83_isr_946180_62.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a3d96fc46b1e5a7d8504fbc0a84e4e2 with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash f44eeb567c9248f95119affc71f159a6
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fccea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 405
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7fc76e6f-d585-4d9e-b3ed-f32eee3ecebc_characterizeImage_946182_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7fc76e6f-d585-4d9e-b3ed-f32eee3ecebc_characterizeImage_946182_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a5fd5f33-1348-4678-8c46-7d339360ed90_isr_946182_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a5fd5f33-1348-4678-8c46-7d339360ed90_isr_946182_39.stderr
parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30c242eb738917c56ebccf1ea3f16c02 with result from task 342
parsl.dataflow.dflow DEBUG: Task 342 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 74163c9ac7ae792e55fe74f3923c8ae5
parsl.dataflow.memoization INFO: Task 343 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 406
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/277842ee-4f53-4a87-867a-ddc20f13ffe9_characterizeImage_946181_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/277842ee-4f53-4a87-867a-ddc20f13ffe9_characterizeImage_946181_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/dd09dc3c-63dc-4365-9dfc-874ffd4e473e_isr_946181_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/dd09dc3c-63dc-4365-9dfc-874ffd4e473e_isr_946181_21.stderr
parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cdadc5d8dc949d5b7855cda750cea0d with result from task 345
parsl.dataflow.dflow DEBUG: Task 345 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 06a0149a0e0a7dceabc4bbe1bc5dc0f2
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ebe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 407
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/b225f401-fd9e-4933-9006-bd1b0dc57d85_characterizeImage_946074_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/b225f401-fd9e-4933-9006-bd1b0dc57d85_characterizeImage_946074_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/dcccff5a-9f8a-4a95-9f6c-efd1195155c6_isr_946074_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/dcccff5a-9f8a-4a95-9f6c-efd1195155c6_isr_946074_50.stderr
parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry feafefa3e84205cc7978c9b9f2b2ba1a with result from task 348
parsl.dataflow.dflow DEBUG: Task 348 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 4b3d80aaaf0287b65af5e1a1bccdb15b
parsl.dataflow.memoization INFO: Task 349 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 408
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/67493d7b-ed84-4557-842a-8c6fe0b6ee32_characterizeImage_946180_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/67493d7b-ed84-4557-842a-8c6fe0b6ee32_characterizeImage_946180_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/090d5f59-40ea-4385-8aab-20a883699e84_isr_946180_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/090d5f59-40ea-4385-8aab-20a883699e84_isr_946180_56.stderr
parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 29080d88610ef2a59102572c77b37b33 with result from task 351
parsl.dataflow.dflow DEBUG: Task 351 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 84f3daea523138cf611a7b6989ca21fe
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 409
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0ba8fa28-f898-475d-a543-661a5d75b02a_characterizeImage_946072_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0ba8fa28-f898-475d-a543-661a5d75b02a_characterizeImage_946072_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/87c631f4-6d58-4662-a12a-67ef1e43df89_isr_946072_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/87c631f4-6d58-4662-a12a-67ef1e43df89_isr_946072_26.stderr
parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2190e68af0ff6b51885f9689c2b5139 with result from task 354
parsl.dataflow.dflow DEBUG: Task 354 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 739575a09e876504e2c87cf7538ae332
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 410
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fab348cd-12ca-4d18-957f-f4a852fa286e_characterizeImage_946074_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fab348cd-12ca-4d18-957f-f4a852fa286e_characterizeImage_946074_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d3f36c0e-1a16-44ca-8ad9-492106cf1c37_isr_946074_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d3f36c0e-1a16-44ca-8ad9-492106cf1c37_isr_946074_44.stderr
parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4e71528c4183533b0110a8c587948a8 with result from task 228
parsl.dataflow.dflow DEBUG: Task 228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 425d65f3c494c3179eb48bc25bea3135
parsl.dataflow.memoization INFO: Task 229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 411
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/10b39b23-a282-48a8-9f52-e9eb9454c857_characterizeImage_946181_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/10b39b23-a282-48a8-9f52-e9eb9454c857_characterizeImage_946181_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d69c93f5-dd03-4afd-9af7-fce3330509be_isr_946181_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d69c93f5-dd03-4afd-9af7-fce3330509be_isr_946181_32.stderr
parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 706ac19756041f7611f7ee65402c938e with result from task 429
parsl.dataflow.dflow DEBUG: Task 429 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 574cab65289e73b66f21d667fed0a3e9
parsl.dataflow.memoization INFO: Task 430 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 412
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/685f34a1-a470-4f84-8836-66f376a7eeee_isr_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/685f34a1-a470-4f84-8836-66f376a7eeee_isr_946181_2.stderr
parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43f048d253ac2504f3c05ea54c89eff7 with result from task 210
parsl.dataflow.dflow DEBUG: Task 210 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 9f127bd8cdaf1c3f3a009eef3d99ee4d
parsl.dataflow.memoization INFO: Task 211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 413
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/34cc8df6-7576-45a0-9901-b0952ab91cc4_characterizeImage_946072_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/34cc8df6-7576-45a0-9901-b0952ab91cc4_characterizeImage_946072_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fbff1017-9f6e-4005-a5d5-2d81abb0a953_isr_946072_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fbff1017-9f6e-4005-a5d5-2d81abb0a953_isr_946072_15.stderr
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18ef1915fccc9534d017d0b429dab6ae with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash f11a3fcb9c4ff6bbd45b7bd693c1dec6
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 414
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8843b2ff-275b-45f7-b6f8-bf9b8290cf05_characterizeImage_946072_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8843b2ff-275b-45f7-b6f8-bf9b8290cf05_characterizeImage_946072_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bfe9b614-8b2b-4c3c-aa51-4dc3e412df6a_isr_946072_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bfe9b614-8b2b-4c3c-aa51-4dc3e412df6a_isr_946072_55.stderr
parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 595da09c18c7c34cc8fc8e2f955645ad with result from task 363
parsl.dataflow.dflow DEBUG: Task 363 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash f5953621bc1e55dff44e9eac6ebda7db
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 415
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3f4df01d-20fe-4f07-b1d7-1c5890055e0e_characterizeImage_946180_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3f4df01d-20fe-4f07-b1d7-1c5890055e0e_characterizeImage_946180_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9eae70a2-1baf-4d7b-b3b5-2d6236f776c5_isr_946180_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9eae70a2-1baf-4d7b-b3b5-2d6236f776c5_isr_946180_37.stderr
parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c16d54db2431f2e0016cf2d969a49de with result from task 357
parsl.dataflow.dflow DEBUG: Task 357 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 358 has memoization hash a65935f7988be0e066e62b3e47e682de
parsl.dataflow.memoization INFO: Task 358 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 416
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/f28ca535-ed4c-402b-9326-713376c40278_characterizeImage_946072_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/f28ca535-ed4c-402b-9326-713376c40278_characterizeImage_946072_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/b32e454b-32ce-4759-a3fb-75be3b5077c6_isr_946072_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/b32e454b-32ce-4759-a3fb-75be3b5077c6_isr_946072_38.stderr
parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 216a24e2f684a0c39e7bb941792abdcf with result from task 327
parsl.dataflow.dflow DEBUG: Task 327 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 5afdcdb0f9b95a01e895e3f49ccba598
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 417
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b3d3a648-d1f0-4fb4-8ab9-b9d66b3e9f1a_characterizeImage_946073_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b3d3a648-d1f0-4fb4-8ab9-b9d66b3e9f1a_characterizeImage_946073_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/81ad036b-5580-415d-9ab2-fa3bbf01d770_isr_946073_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/81ad036b-5580-415d-9ab2-fa3bbf01d770_isr_946073_15.stderr
parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7de70fcb220d2a588b0a924e744cbbe9 with result from task 330
parsl.dataflow.dflow DEBUG: Task 330 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 9259d1238282fda1b1cce336c051e829
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 418
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1c1174bd-eafd-4c07-98f0-d11b82ae8d83_characterizeImage_946072_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1c1174bd-eafd-4c07-98f0-d11b82ae8d83_characterizeImage_946072_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ad19903d-f067-4175-a652-b94755ae9eb1_isr_946072_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ad19903d-f067-4175-a652-b94755ae9eb1_isr_946072_51.stderr
parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2823ee5692121c01f36612147b5d9e1f with result from task 189
parsl.dataflow.dflow DEBUG: Task 189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 78c78c2b475a2bac82eeb90515dd1063
parsl.dataflow.memoization INFO: Task 190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 419
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/37301de0-5228-4184-ab80-9f54dc7c410a_characterizeImage_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/37301de0-5228-4184-ab80-9f54dc7c410a_characterizeImage_946181_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bcbd1ee5-8f5b-495a-be47-58a11cc2269b_isr_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bcbd1ee5-8f5b-495a-be47-58a11cc2269b_isr_946181_31.stderr
parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec8f95d64f3540d7bf87b10a64b043bc with result from task 360
parsl.dataflow.dflow DEBUG: Task 360 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 090b3341c3c13a62b40b2772fc379550
parsl.dataflow.memoization INFO: Task 361 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 420
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b6d4662e-3c43-44c9-a36f-edaaadaee3f7_characterizeImage_946180_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b6d4662e-3c43-44c9-a36f-edaaadaee3f7_characterizeImage_946180_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/932424ee-12e7-443a-9e25-d546e910c0d0_isr_946180_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/932424ee-12e7-443a-9e25-d546e910c0d0_isr_946180_12.stderr
parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60543a566b72966b438220ffc41a705f with result from task 423
parsl.dataflow.dflow DEBUG: Task 423 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 424 has memoization hash e4aace4225735e9b47dc3f3a15396d24
parsl.dataflow.memoization INFO: Task 424 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 421
parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/07e5f42d-acbf-4bda-801e-27ae5f9169d2_characterizeImage_946182_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/07e5f42d-acbf-4bda-801e-27ae5f9169d2_characterizeImage_946182_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58d977e2-6e16-4662-ae76-69d07cb687f9_isr_946182_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58d977e2-6e16-4662-ae76-69d07cb687f9_isr_946182_13.stderr
parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b707eaa4a4c6dfced2d4242dfa945cc with result from task 318
parsl.dataflow.dflow DEBUG: Task 318 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 2c508034f9489522dbbdfe265317e64c
parsl.dataflow.memoization INFO: Task 319 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 422
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/3bf4b117-1a32-4455-b84e-a947139ae66a_characterizeImage_946182_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/3bf4b117-1a32-4455-b84e-a947139ae66a_characterizeImage_946182_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f76d25c4-19b5-44b1-a272-57a86585b4af_isr_946182_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f76d25c4-19b5-44b1-a272-57a86585b4af_isr_946182_62.stderr
parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 375c4f527f0c6ef9bfa30272e95757a3 with result from task 321
parsl.dataflow.dflow DEBUG: Task 321 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 322 has memoization hash ff943726815c34d9ce1d912ecc6d51ce
parsl.dataflow.memoization INFO: Task 322 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 423
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/13ca9d75-e554-4348-a5a0-774a1c4fc205_characterizeImage_946180_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/13ca9d75-e554-4348-a5a0-774a1c4fc205_characterizeImage_946180_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b7ef4c01-3385-4ed2-a0e5-89441678f3b7_isr_946180_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b7ef4c01-3385-4ed2-a0e5-89441678f3b7_isr_946180_3.stderr
parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51fdccecdec2da4acb5ad8123c1be5a9 with result from task 432
parsl.dataflow.dflow DEBUG: Task 432 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 433 has memoization hash c1a7c0d10550cfb533940261c25b04ae
parsl.dataflow.memoization INFO: Task 433 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 424
parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7d1b3-0433-4872-a5c5-6360481de193_characterizeImage_946073_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7d1b3-0433-4872-a5c5-6360481de193_characterizeImage_946073_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/2da0e428-eadb-4c04-8671-9dedd0295304_isr_946073_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/2da0e428-eadb-4c04-8671-9dedd0295304_isr_946073_19.stderr
parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 926416b4d2d056eb05b4a2ed5a946187 with result from task 408
parsl.dataflow.dflow DEBUG: Task 408 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 42d77eb14a3a84b3544107491ef2a6fb
parsl.dataflow.memoization INFO: Task 409 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 425
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3c103a23-ae40-435d-a860-2bcc75f25010_isr_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3c103a23-ae40-435d-a860-2bcc75f25010_isr_946182_2.stderr
parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd8ded10dd489bbe3ac30afcedac21b7 with result from task 192
parsl.dataflow.dflow DEBUG: Task 192 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 193 has memoization hash efe4e8018780b7c926a76b62b9c1fffb
parsl.dataflow.memoization INFO: Task 193 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 426
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0e2ce6c5-7bd4-46ab-8791-41e4bf8e813a_characterizeImage_946182_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0e2ce6c5-7bd4-46ab-8791-41e4bf8e813a_characterizeImage_946182_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8a51dbc0-59b3-42dc-9d62-255fbddb829a_isr_946182_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8a51dbc0-59b3-42dc-9d62-255fbddb829a_isr_946182_23.stderr
parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1647105b345b17141d3e1eb1d3e73ea9 with result from task 378
parsl.dataflow.dflow DEBUG: Task 378 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 379 has memoization hash d649b0637dee0ab81535c416bdd40a72
parsl.dataflow.memoization INFO: Task 379 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 427
parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecd2800c-8c72-4bf9-a412-0f16bdf19594_characterizeImage_946073_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecd2800c-8c72-4bf9-a412-0f16bdf19594_characterizeImage_946073_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b8e30b77-ca47-445f-8875-e3a4798dafc0_isr_946073_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b8e30b77-ca47-445f-8875-e3a4798dafc0_isr_946073_45.stderr
parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea3b7eb080269d7c403999bb48f3d3e2 with result from task 375
parsl.dataflow.dflow DEBUG: Task 375 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 376 has memoization hash e72835816af6ed883c10843d976ac6a1
parsl.dataflow.memoization INFO: Task 376 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 428
parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/56e00260-8f2f-4e62-841d-091da0f628ce_characterizeImage_946180_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/56e00260-8f2f-4e62-841d-091da0f628ce_characterizeImage_946180_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b2ec709-fbd9-4e1b-b38c-5f7b0f9a71fe_isr_946180_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b2ec709-fbd9-4e1b-b38c-5f7b0f9a71fe_isr_946180_8.stderr
parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e309069bbf92c3d1f56ecc03b2418ca4 with result from task 336
parsl.dataflow.dflow DEBUG: Task 336 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 180edca4e1731bdc2a7c048843e0c992
parsl.dataflow.memoization INFO: Task 337 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 429
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50832a28-bcf2-4385-9b00-747bdb43231a_characterizeImage_946181_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50832a28-bcf2-4385-9b00-747bdb43231a_characterizeImage_946181_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/eca15dec-3612-40af-a634-381af6f394de_isr_946181_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/eca15dec-3612-40af-a634-381af6f394de_isr_946181_13.stderr
parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5440c270e641eb9603efe786e5e7f92a with result from task 366
parsl.dataflow.dflow DEBUG: Task 366 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 367 has memoization hash df7062295a50ff85c6672868beac708a
parsl.dataflow.memoization INFO: Task 367 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 430
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/65e9533b-d25a-48da-95f6-f5816579e56b_characterizeImage_946074_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/65e9533b-d25a-48da-95f6-f5816579e56b_characterizeImage_946074_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/cbda7158-21b5-45a2-ba39-1437971d5587_isr_946074_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/cbda7158-21b5-45a2-ba39-1437971d5587_isr_946074_9.stderr
parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 300a8f9f2549d782243d5bf484d6d53b with result from task 333
parsl.dataflow.dflow DEBUG: Task 333 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 175be84982ada72c21568a9378530271
parsl.dataflow.memoization INFO: Task 334 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 431
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3e9d4392-4196-458c-a2ce-c636a565ccbe_characterizeImage_946180_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3e9d4392-4196-458c-a2ce-c636a565ccbe_characterizeImage_946180_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/198ae5a7-70de-4e22-9aac-7c9d9cade973_isr_946180_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/198ae5a7-70de-4e22-9aac-7c9d9cade973_isr_946180_31.stderr
parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca9d1e8c5db60a22b78b4ff334de672a with result from task 369
parsl.dataflow.dflow DEBUG: Task 369 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 8430c6161fb63ce7bc40c451a0de2583
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 432
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/22281e2b-fcf5-47ef-a400-caec837424af_characterizeImage_946074_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/22281e2b-fcf5-47ef-a400-caec837424af_characterizeImage_946074_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a45a54fe-a897-4fc5-b0de-de48457fac79_isr_946074_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a45a54fe-a897-4fc5-b0de-de48457fac79_isr_946074_26.stderr
parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e4c0b4a2241fda7c3cee3edf21ca6e0 with result from task 414
parsl.dataflow.dflow DEBUG: Task 414 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 9a1ccb9750e1aeaff8842bb94ae47430
parsl.dataflow.memoization INFO: Task 415 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 433
parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/02211b09-f547-4bbc-afd1-bcafa89864c4_characterizeImage_946180_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/02211b09-f547-4bbc-afd1-bcafa89864c4_characterizeImage_946180_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ae9b6ebf-b1b1-4b29-883e-ad634e03a19f_isr_946180_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ae9b6ebf-b1b1-4b29-883e-ad634e03a19f_isr_946180_17.stderr
parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ee6ff9e24a84e41eb508c987486ec2b with result from task 402
parsl.dataflow.dflow DEBUG: Task 402 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 89aaa57a3f36bbad0d0a33a3b859baf7
parsl.dataflow.memoization INFO: Task 403 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 434
parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/1709020b-7329-4630-84ad-4956984764f4_characterizeImage_946074_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/1709020b-7329-4630-84ad-4956984764f4_characterizeImage_946074_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8acae5a7-8ca7-4f68-ae0f-08e7a4ebdf9e_isr_946074_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8acae5a7-8ca7-4f68-ae0f-08e7a4ebdf9e_isr_946074_17.stderr
parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5054b9fc301b7afbb1bb2013af2a5de with result from task 213
parsl.dataflow.dflow DEBUG: Task 213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 214 has memoization hash d0bd8697054fce0923aef190c1f385cc
parsl.dataflow.memoization INFO: Task 214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 435
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d5b3e8fb-21bd-4e8b-8870-d13325eaaf97_characterizeImage_946180_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d5b3e8fb-21bd-4e8b-8870-d13325eaaf97_characterizeImage_946180_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e347a3a3-0895-4ef3-b5ad-db3337a2e1eb_isr_946180_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e347a3a3-0895-4ef3-b5ad-db3337a2e1eb_isr_946180_9.stderr
parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20628bec6ac46f43ec6eb86f9530e5ba with result from task 198
parsl.dataflow.dflow DEBUG: Task 198 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 9cd45a7ca4a9692db68dae392719568d
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 436
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c4dc3a13-9606-4b5d-ac1d-c904da17d158_characterizeImage_946181_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c4dc3a13-9606-4b5d-ac1d-c904da17d158_characterizeImage_946181_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82a07f45-93fb-40fe-ba9e-4b77c3fb3197_isr_946181_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82a07f45-93fb-40fe-ba9e-4b77c3fb3197_isr_946181_5.stderr
parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66948aa662ad0db74f32dbd4422d3cdf with result from task 372
parsl.dataflow.dflow DEBUG: Task 372 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 373 has memoization hash af6ffe4dfca59c882bc2852b500e7751
parsl.dataflow.memoization INFO: Task 373 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 437
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/126109d9-74e6-419f-8923-c1993153b241_characterizeImage_946182_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/126109d9-74e6-419f-8923-c1993153b241_characterizeImage_946182_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb981c36-e331-48c1-8084-ad6bae302491_isr_946182_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb981c36-e331-48c1-8084-ad6bae302491_isr_946182_24.stderr
parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab73ef9dcc19cfb8d600483f05ef3148 with result from task 219
parsl.dataflow.dflow DEBUG: Task 219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 2daa3553a5930b5ef916193a0cecd964
parsl.dataflow.memoization INFO: Task 220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 438
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c941b2a8-046d-49b4-9f82-89df9237a447_characterizeImage_946181_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c941b2a8-046d-49b4-9f82-89df9237a447_characterizeImage_946181_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c06ed5be-a5e4-4db4-95de-2b1a0c50cbfb_isr_946181_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c06ed5be-a5e4-4db4-95de-2b1a0c50cbfb_isr_946181_35.stderr
parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6924149b001fbc15e0dd4cba9aedf364 with result from task 441
parsl.dataflow.dflow DEBUG: Task 441 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 442 has memoization hash c6d5b8322957fd916c76b69a5e95bd6b
parsl.dataflow.memoization INFO: Task 442 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 439
parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e6e08228-697a-4fab-98b5-3ad893c8fd36_characterizeImage_946181_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e6e08228-697a-4fab-98b5-3ad893c8fd36_characterizeImage_946181_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6b8c4428-d70a-4dd2-a838-eab61204ec8c_isr_946181_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6b8c4428-d70a-4dd2-a838-eab61204ec8c_isr_946181_60.stderr
parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f12bbe5a7dfb5aecbb8cc475e03a4a9 with result from task 222
parsl.dataflow.dflow DEBUG: Task 222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 0e61db68948d3e95ff656fec8acce2c3
parsl.dataflow.memoization INFO: Task 223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 440
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3f87ce3e-0000-457e-bd24-f69329b5d1d5_characterizeImage_946181_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3f87ce3e-0000-457e-bd24-f69329b5d1d5_characterizeImage_946181_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2a78194a-ab57-4b52-a55e-ddbebcb01f39_isr_946181_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2a78194a-ab57-4b52-a55e-ddbebcb01f39_isr_946181_28.stderr
parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26b7947d4e01d804e885ffbff005e5ca with result from task 438
parsl.dataflow.dflow DEBUG: Task 438 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 5c13b606d337b3acc77ba5df5a9f51af
parsl.dataflow.memoization INFO: Task 439 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 441
parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cf659be7-e4c8-41bd-830f-8ee86cfded05_characterizeImage_946182_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cf659be7-e4c8-41bd-830f-8ee86cfded05_characterizeImage_946182_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/de7dce0b-1ad5-4896-ac73-4e73a27efe4d_isr_946182_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/de7dce0b-1ad5-4896-ac73-4e73a27efe4d_isr_946182_17.stderr
parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f3139f59b33cfb7e27e0533e34d529f2 with result from task 435
parsl.dataflow.dflow DEBUG: Task 435 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 9fa00e6e324e5e34dd0b3165b18f5ca2
parsl.dataflow.memoization INFO: Task 436 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 442
parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/04392313-25d2-4e7a-86a7-c040b6b2c5e9_characterizeImage_946074_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/04392313-25d2-4e7a-86a7-c040b6b2c5e9_characterizeImage_946074_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/f02f53a9-5347-4463-b6a7-7eeb59cd67db_isr_946074_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/f02f53a9-5347-4463-b6a7-7eeb59cd67db_isr_946074_14.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 598e3cc167dd0096b5206389257c8017 with result from task 396
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow DEBUG: Task 396 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 397 has memoization hash c3f31ec69f1619082a4efb6fc0124e06
parsl.dataflow.memoization INFO: Task 397 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 443
parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/4f4655bd-21c5-4580-88ff-55bb602072d7_characterizeImage_946181_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/4f4655bd-21c5-4580-88ff-55bb602072d7_characterizeImage_946181_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/be8c82c7-401a-4127-bde5-2f9ad37be993_isr_946181_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/be8c82c7-401a-4127-bde5-2f9ad37be993_isr_946181_51.stderr
parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c840559757f9ffd0fc2a896cb47295d0 with result from task 231
parsl.dataflow.dflow DEBUG: Task 231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 232 has memoization hash d7a7bc282c91a52779e4960956b9dcf9
parsl.dataflow.memoization INFO: Task 232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 444
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/849d1649-1873-4b73-8ab5-de3f6bf39320_characterizeImage_946182_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/849d1649-1873-4b73-8ab5-de3f6bf39320_characterizeImage_946182_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a1cf5248-bb8d-4759-9829-b433ac987ef3_isr_946182_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a1cf5248-bb8d-4759-9829-b433ac987ef3_isr_946182_22.stderr
parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc9002303cb9d8ab5ff636355d750caa with result from task 216
parsl.dataflow.dflow DEBUG: Task 216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 23825408a6acbc5cf5271b91e255587e
parsl.dataflow.memoization INFO: Task 217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 445
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/f709fd05-f342-4806-b594-aa9a967f3afa_characterizeImage_946180_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/f709fd05-f342-4806-b594-aa9a967f3afa_characterizeImage_946180_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/543105da-58b2-40de-8150-de2b9f66a2c1_isr_946180_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/543105da-58b2-40de-8150-de2b9f66a2c1_isr_946180_39.stderr
parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2179b4b6af66341d6458da472718381 with result from task 339
parsl.dataflow.dflow DEBUG: Task 339 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 16354dec216b4cce6a3c7febf51adb79
parsl.dataflow.memoization INFO: Task 340 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 446
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fcb7cd7c-ee47-4860-9f30-082eb7bb18be_characterizeImage_946180_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fcb7cd7c-ee47-4860-9f30-082eb7bb18be_characterizeImage_946180_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7ee4f318-f71e-4303-b2a2-bd7df1fcca5c_isr_946180_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7ee4f318-f71e-4303-b2a2-bd7df1fcca5c_isr_946180_55.stderr
parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 800a3e3c0be7d0d59e4292779c71f0ad with result from task 324
parsl.dataflow.dflow DEBUG: Task 324 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 3b7ed911597abb6271f01a447a3b09a7
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 447
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/27e87bb0-c388-42ad-a680-ee1af40cecec_characterizeImage_946180_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/27e87bb0-c388-42ad-a680-ee1af40cecec_characterizeImage_946180_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/34d0c732-cce5-4c71-ba30-04c3ad034ede_isr_946180_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/34d0c732-cce5-4c71-ba30-04c3ad034ede_isr_946180_29.stderr
parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5a1c4f2864e727afdd55648f127e5f with result from task 381
parsl.dataflow.dflow DEBUG: Task 381 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 382 has memoization hash c9779ebb141d7a3958651c76b1e96a89
parsl.dataflow.memoization INFO: Task 382 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 448
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e782443c-88fc-4a3f-8301-77e1d2c1ff51_characterizeImage_946073_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e782443c-88fc-4a3f-8301-77e1d2c1ff51_characterizeImage_946073_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/28ffa85f-f5c0-45ba-90bb-c33a24a0050d_isr_946073_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/28ffa85f-f5c0-45ba-90bb-c33a24a0050d_isr_946073_36.stderr
parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac7a5160fa775cff90b9ce7450394215 with result from task 399
parsl.dataflow.dflow DEBUG: Task 399 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 400 has memoization hash aec1e842c386684f4b120c666de5b696
parsl.dataflow.memoization INFO: Task 400 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 449
parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7b22a60-f162-4ff3-8e30-292a3e977dd5_characterizeImage_946181_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7b22a60-f162-4ff3-8e30-292a3e977dd5_characterizeImage_946181_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/11066b3a-e737-4ab3-b814-b2d21f68d560_isr_946181_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/11066b3a-e737-4ab3-b814-b2d21f68d560_isr_946181_50.stderr
parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4221bfc40ae340e3512208436094b6b with result from task 405
parsl.dataflow.dflow DEBUG: Task 405 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 3163036c5ba7d12779ffcc1396586351
parsl.dataflow.memoization INFO: Task 406 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 450
parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/921b14b8-b0c3-4a1f-8c01-f63bcd18af63_characterizeImage_946182_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/921b14b8-b0c3-4a1f-8c01-f63bcd18af63_characterizeImage_946182_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6ef07143-7bec-4116-a19a-2a6e9db28a37_isr_946182_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6ef07143-7bec-4116-a19a-2a6e9db28a37_isr_946182_48.stderr
parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2095af55dc12a4c85fc2c0734989e448 with result from task 384
parsl.dataflow.dflow DEBUG: Task 384 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 7c20033bbddfabbbd29234f0d6856fe9
parsl.dataflow.memoization INFO: Task 385 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 451
parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/50be2ad6-dc8b-44c3-8758-0800e69de90e_characterizeImage_946073_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/50be2ad6-dc8b-44c3-8758-0800e69de90e_characterizeImage_946073_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6e447186-8036-43f1-8ba9-e73ce5fdb348_isr_946073_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6e447186-8036-43f1-8ba9-e73ce5fdb348_isr_946073_60.stderr
parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcf4782a0ed40da29b938f4aeb0f0999 with result from task 195
parsl.dataflow.dflow DEBUG: Task 195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 66eec8c889dc6f9f97e78823c36dae08
parsl.dataflow.memoization INFO: Task 196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 452
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f4175c90-75a0-49e4-a71d-48b54bbb2d12_characterizeImage_946073_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f4175c90-75a0-49e4-a71d-48b54bbb2d12_characterizeImage_946073_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/7addc496-aaba-4ba8-8d1c-15feabbe1443_isr_946073_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/7addc496-aaba-4ba8-8d1c-15feabbe1443_isr_946073_14.stderr
parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09d4e1e7e5aad0af5c541ed1b0da0577 with result from task 387
parsl.dataflow.dflow DEBUG: Task 387 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 388 has memoization hash a946235fb0ad5441ce084935609e3293
parsl.dataflow.memoization INFO: Task 388 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 453
parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/07b287b7-c36b-4d22-9c6c-94a2b53ff981_characterizeImage_946072_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/07b287b7-c36b-4d22-9c6c-94a2b53ff981_characterizeImage_946072_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/42d694db-a714-44b2-98f4-021411463551_isr_946072_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/42d694db-a714-44b2-98f4-021411463551_isr_946072_57.stderr
parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0e9922210a954c7cafde3eebc39764f with result from task 444
parsl.dataflow.dflow DEBUG: Task 444 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 445 has memoization hash b007e7c42c65b54e5f18ec773d20da03
parsl.dataflow.memoization INFO: Task 445 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 454
parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/766ef5b8-ed08-405a-a05f-06db471be75c_characterizeImage_946180_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/766ef5b8-ed08-405a-a05f-06db471be75c_characterizeImage_946180_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/908b04bb-78b8-4f4e-addc-760c057a78a9_isr_946180_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/908b04bb-78b8-4f4e-addc-760c057a78a9_isr_946180_53.stderr
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56f50180de5a7d78eb6a5056e2226112 with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash bc3955a6752b8e46a003a2446dbe4d5b
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 455
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/542783a3-58d3-47a7-9e0e-451e68c12c30_characterizeImage_946181_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/542783a3-58d3-47a7-9e0e-451e68c12c30_characterizeImage_946181_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6ae940f9-4a5e-4872-892d-5ba16808bd0c_isr_946181_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/6ae940f9-4a5e-4872-892d-5ba16808bd0c_isr_946181_14.stderr
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c15571b473c4fdc013a7cb3662235f9 with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash e3ff768b0c4c098da97ea6f40685b5e8
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 456
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/882089e9-028e-45b2-a812-1a3442e5bae8_characterizeImage_946182_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/882089e9-028e-45b2-a812-1a3442e5bae8_characterizeImage_946182_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/40bffd92-6e31-4e85-bb79-dc4fcbb6d485_isr_946182_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/40bffd92-6e31-4e85-bb79-dc4fcbb6d485_isr_946182_43.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 085c1fad3b86743c97717affc8c40b40 with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 55260c8ff7606420a121d525265b65a6
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 457
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/063a5d36-3355-4d30-b09c-221e4f424059_characterizeImage_946180_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/063a5d36-3355-4d30-b09c-221e4f424059_characterizeImage_946180_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3dd11fc1-5698-4fc2-9f5b-4a044c4dd2d9_isr_946180_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3dd11fc1-5698-4fc2-9f5b-4a044c4dd2d9_isr_946180_48.stderr
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f50fc50737ae075e3f07a06220446bf8 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 11bbe06260aeba52be42154fa6f8e96a
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 458
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/70e3c3b5-5695-4850-a7ea-f4a00e122258_characterizeImage_946073_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/70e3c3b5-5695-4850-a7ea-f4a00e122258_characterizeImage_946073_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e8bb882b-5184-4bab-8f01-c679b5829b5b_isr_946073_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e8bb882b-5184-4bab-8f01-c679b5829b5b_isr_946073_48.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0de16d9b862df72bbf7dddc40b84bbbd with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash f030bc6dd091b1be8b54e9cbae44b5bc
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 459
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2e686eb1-55f2-473b-9aeb-12b777c57b7d_characterizeImage_946182_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2e686eb1-55f2-473b-9aeb-12b777c57b7d_characterizeImage_946182_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ff9b1499-6fb7-41fe-b2b8-d1803346188b_isr_946182_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ff9b1499-6fb7-41fe-b2b8-d1803346188b_isr_946182_50.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2263a942ec9f719a87402737541c5c23 with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 95de1123a20efca3aa9f5c68b5ab5beb
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 460
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a2df69b9-bec6-4c89-96f9-37203b9580ac_characterizeImage_946182_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a2df69b9-bec6-4c89-96f9-37203b9580ac_characterizeImage_946182_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/fb69275d-2211-438f-abe9-2069b1fa0bf2_isr_946182_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/fb69275d-2211-438f-abe9-2069b1fa0bf2_isr_946182_51.stderr
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06b3f8e99c47c1ec2ed0c23c7e1ef161 with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash eecca9237bcc77c2aaba7e94632a19f0
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 461
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/46bc9375-fee6-4038-b442-a5679437d612_characterizeImage_946074_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/46bc9375-fee6-4038-b442-a5679437d612_characterizeImage_946074_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/114f42e1-a86e-45a0-aadd-46ecf1680bb8_isr_946074_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/114f42e1-a86e-45a0-aadd-46ecf1680bb8_isr_946074_48.stderr
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f7f78fc82995ca55f01f7ce5dd88241 with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 48f5e8d3cfe003003e84673802be5039
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 462
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/68ce8470-b918-4ce9-96da-a2c5bbf083d1_characterizeImage_946073_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/68ce8470-b918-4ce9-96da-a2c5bbf083d1_characterizeImage_946073_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d587ba4b-827a-4185-8ef2-2d9ac5708c47_isr_946073_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d587ba4b-827a-4185-8ef2-2d9ac5708c47_isr_946073_20.stderr
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6485744568fe8d2ad7d0f402235350ed with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash df97f203a999e0ae40b0d54e89cdb380
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 463
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d0261385-d7b5-4329-b587-6bf50d30054a_characterizeImage_946181_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d0261385-d7b5-4329-b587-6bf50d30054a_characterizeImage_946181_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/0de60304-6635-40c6-b829-768f79591285_isr_946181_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/0de60304-6635-40c6-b829-768f79591285_isr_946181_55.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64c675779265eabc48b2de548e54b94c with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b86defd228a3069f348f404ab1201e08
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 464
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b7ee437-7916-4e8b-ac62-b211a224750a_characterizeImage_946181_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b7ee437-7916-4e8b-ac62-b211a224750a_characterizeImage_946181_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/58a3f5e7-6687-4199-ab63-2d236cb9541c_isr_946181_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/58a3f5e7-6687-4199-ab63-2d236cb9541c_isr_946181_46.stderr
parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96612c0e248e2a4bb05a2c5b59e173bf with result from task 201
parsl.dataflow.dflow DEBUG: Task 201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 40f019b89443af7b4070cf121dcf7718
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 465
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ad93d17e-4dc2-4285-a633-1be431e45509_characterizeImage_946073_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ad93d17e-4dc2-4285-a633-1be431e45509_characterizeImage_946073_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b165e9ba-0238-44c5-a051-a7bb0e28ab60_isr_946073_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b165e9ba-0238-44c5-a051-a7bb0e28ab60_isr_946073_50.stderr
parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f5bc58ed70a8df58678830d0822ac85 with result from task 225
parsl.dataflow.dflow DEBUG: Task 225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 439a075cf4458aec7daed523980793da
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 466
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/db394b94-2e64-4fac-868c-a30f5bd62eec_characterizeImage_946074_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/db394b94-2e64-4fac-868c-a30f5bd62eec_characterizeImage_946074_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/aaa38cd5-74f0-43fe-8f49-9c4d118541bb_isr_946074_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/aaa38cd5-74f0-43fe-8f49-9c4d118541bb_isr_946074_56.stderr
parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 202423979ef6f93a2321c77b11b159c7 with result from task 1038
parsl.dataflow.dflow DEBUG: Task 1038 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash e64aa16172916a401160ddcb62cdd6dd
parsl.dataflow.memoization INFO: Task 1039 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 467
parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd678ea9-bb1a-4db3-8f71-30f75c92c654_characterizeImage_946181_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd678ea9-bb1a-4db3-8f71-30f75c92c654_characterizeImage_946181_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/441f4339-7b68-4d53-8726-0b401b433ce2_isr_946181_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/441f4339-7b68-4d53-8726-0b401b433ce2_isr_946181_16.stderr
parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e76eb410777fc2a34728f8709e70d88 with result from task 879
parsl.dataflow.dflow DEBUG: Task 879 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 53ebcf5ba7ee5cac187fb2d2ec20f8a3
parsl.dataflow.memoization INFO: Task 880 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 468
parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7e997f3d-5657-4727-afbd-f71b57fad166_characterizeImage_946180_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7e997f3d-5657-4727-afbd-f71b57fad166_characterizeImage_946180_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fed15b00-d53f-478e-909e-60d5f6db31c0_isr_946180_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fed15b00-d53f-478e-909e-60d5f6db31c0_isr_946180_14.stderr
parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff38d667c190e5533e751ee60b0fca31 with result from task 207
parsl.dataflow.dflow DEBUG: Task 207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 79b0495cda5a3805e2dae5b1bc02204e
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 469
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/50a6ceab-81d4-43fd-b170-91bf84146ed8_characterizeImage_946182_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/50a6ceab-81d4-43fd-b170-91bf84146ed8_characterizeImage_946182_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1170d194-9bad-47c1-a16d-585d31b40bae_isr_946182_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1170d194-9bad-47c1-a16d-585d31b40bae_isr_946182_36.stderr
parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcfa7983bf15c0c74a1102d779744a5b with result from task 819
parsl.dataflow.dflow DEBUG: Task 819 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 278ac09c1d36596effc59ea2be5831d4
parsl.dataflow.memoization INFO: Task 820 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 470
parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e3cfbf29-b437-4d26-a5b3-60086814f200_characterizeImage_946074_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e3cfbf29-b437-4d26-a5b3-60086814f200_characterizeImage_946074_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/25694540-773d-4240-b83b-7ca2ca26b401_isr_946074_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/25694540-773d-4240-b83b-7ca2ca26b401_isr_946074_27.stderr
parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e059761e5407ec7854ccc0d9ae3f3d6 with result from task 810
parsl.dataflow.dflow DEBUG: Task 810 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 36825db7c77a6fb073cb0b38ea622562
parsl.dataflow.memoization INFO: Task 811 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 471
parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b4209ad1-aef6-41d7-bd15-2be7a4745bd0_characterizeImage_946180_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b4209ad1-aef6-41d7-bd15-2be7a4745bd0_characterizeImage_946180_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/dead9ac2-b5d5-4ce1-bfb9-28acc89d66a5_isr_946180_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/dead9ac2-b5d5-4ce1-bfb9-28acc89d66a5_isr_946180_15.stderr
parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96a6309014ae3c639538f2dc279d119a with result from task 729
parsl.dataflow.dflow DEBUG: Task 729 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 730 has memoization hash af2d3b938b229f7b9f941e3036591d87
parsl.dataflow.memoization INFO: Task 730 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 472
parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bfa1bf1f-324c-43c2-aea6-873a082011fd_characterizeImage_946182_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bfa1bf1f-324c-43c2-aea6-873a082011fd_characterizeImage_946182_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1cdf86b6-ce26-493c-a2a0-021fe3b13b9d_isr_946182_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/1cdf86b6-ce26-493c-a2a0-021fe3b13b9d_isr_946182_25.stderr
parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 646f4670f4704516ab9a46ae362748db with result from task 714
parsl.dataflow.dflow DEBUG: Task 714 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 6674d81e806488cc005974102b2a5d8b
parsl.dataflow.memoization INFO: Task 715 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 473
parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/420dccb1-5e65-41e3-a764-64e8528e1378_characterizeImage_946073_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/420dccb1-5e65-41e3-a764-64e8528e1378_characterizeImage_946073_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47ef7f8f-4eb0-4836-a5ed-3b4f29097b82_isr_946073_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47ef7f8f-4eb0-4836-a5ed-3b4f29097b82_isr_946073_3.stderr
parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bac5ff2f5625a154716176d18cb769c2 with result from task 723
parsl.dataflow.dflow DEBUG: Task 723 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 3cdecbaabe8959dc9be1b6f7ee1b80cb
parsl.dataflow.memoization INFO: Task 724 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 474
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9cc3c20b-d748-41e9-8275-e17147b12621_characterizeImage_946181_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9cc3c20b-d748-41e9-8275-e17147b12621_characterizeImage_946181_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bb5105c1-7f1a-43f0-a673-c9dd7cc695c0_isr_946181_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bb5105c1-7f1a-43f0-a673-c9dd7cc695c0_isr_946181_18.stderr
parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56f26d8b6d780b48daa5a7074f94eb33 with result from task 711
parsl.dataflow.dflow DEBUG: Task 711 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 1d9f419793b2a892f487ffedabca914d
parsl.dataflow.memoization INFO: Task 712 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 475
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/6fd98e3f-49ca-4768-8a44-6861de632b21_characterizeImage_946074_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/6fd98e3f-49ca-4768-8a44-6861de632b21_characterizeImage_946074_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/14719507-defe-4a2a-af61-a98b0d450bea_isr_946074_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/14719507-defe-4a2a-af61-a98b0d450bea_isr_946074_21.stderr
parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9adedb991c88c93a78742f39d4a2cbe2 with result from task 720
parsl.dataflow.dflow DEBUG: Task 720 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 721 has memoization hash aca6a23fc6a7c7b7080a4256f092e56d
parsl.dataflow.memoization INFO: Task 721 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 476
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ac79f36c-7389-4fe3-9f9e-d1ba2c3077d2_characterizeImage_946073_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ac79f36c-7389-4fe3-9f9e-d1ba2c3077d2_characterizeImage_946073_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dbb60a14-2224-459a-ab6c-1521ab61b068_isr_946073_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dbb60a14-2224-459a-ab6c-1521ab61b068_isr_946073_13.stderr
parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1618885260e3670a1272bcf999382c3 with result from task 846
parsl.dataflow.dflow DEBUG: Task 846 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 063fbc5b8951219ee24041efdb16b3e7
parsl.dataflow.memoization INFO: Task 847 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 477
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cd06ad3a-505f-4fc2-8b0e-ef53749be83b_characterizeImage_946072_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cd06ad3a-505f-4fc2-8b0e-ef53749be83b_characterizeImage_946072_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/0eb66183-6b81-44e2-906d-4b839253272c_isr_946072_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/0eb66183-6b81-44e2-906d-4b839253272c_isr_946072_33.stderr
parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 462d514b9d7dc63947c0109525975952 with result from task 234
parsl.dataflow.dflow DEBUG: Task 234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 97670852ae2c8fbcd5d5ad6406a9c349
parsl.dataflow.memoization INFO: Task 235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 478
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/edfbf97e-c3c1-451b-8ab3-b79694d159f9_characterizeImage_946072_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/edfbf97e-c3c1-451b-8ab3-b79694d159f9_characterizeImage_946072_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3dfd289-babb-4642-92b4-92d99f5f17a7_isr_946072_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3dfd289-babb-4642-92b4-92d99f5f17a7_isr_946072_3.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba5b26490ba1c3925a897a3e880134cd with result from task 204
parsl.dataflow.dflow DEBUG: Task 204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 84254d650f673c56d826400b1fe214c3
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fceac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 479
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e1b422ba-5dda-4aa4-839f-308535753f22_characterizeImage_946180_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e1b422ba-5dda-4aa4-839f-308535753f22_characterizeImage_946180_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4204b4ce-6686-4a6c-b819-d8f9c88bb648_isr_946180_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4204b4ce-6686-4a6c-b819-d8f9c88bb648_isr_946180_59.stderr
parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fa2703b8540ccd69cb871a2a791977c with result from task 732
parsl.dataflow.dflow DEBUG: Task 732 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 33c0894883018b1b6b00a802c495a808
parsl.dataflow.memoization INFO: Task 733 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 480
parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c6be59d9-fb3d-49c5-812b-401d5450925f_characterizeImage_946180_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c6be59d9-fb3d-49c5-812b-401d5450925f_characterizeImage_946180_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fe568b7b-fc62-4ad1-b3aa-3688d8db36c1_isr_946180_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/fe568b7b-fc62-4ad1-b3aa-3688d8db36c1_isr_946180_41.stderr
parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c9151a10615af2b425efcdf14f29ea2 with result from task 1095
parsl.dataflow.dflow DEBUG: Task 1095 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 1ad505255f365608a207cf7fea2b8df5
parsl.dataflow.memoization INFO: Task 1096 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 481
parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/84967d92-62b0-4699-82ef-9fb2909b1708_characterizeImage_946182_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/84967d92-62b0-4699-82ef-9fb2909b1708_characterizeImage_946182_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/717f4f1f-a0e3-4828-a4b2-8e0a5a49fe7b_isr_946182_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/717f4f1f-a0e3-4828-a4b2-8e0a5a49fe7b_isr_946182_27.stderr
parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1469e11fdbc3572ebb4c8dfd22e52be with result from task 816
parsl.dataflow.dflow DEBUG: Task 816 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 817 has memoization hash c9664f1458ee8899d6167cbc01521272
parsl.dataflow.memoization INFO: Task 817 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 482
parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a15d7b41-e6cc-4bc2-a5b8-778e2545eaef_characterizeImage_946073_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a15d7b41-e6cc-4bc2-a5b8-778e2545eaef_characterizeImage_946073_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ddb71e9a-e890-4098-a4d0-cbcc37f7f905_isr_946073_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ddb71e9a-e890-4098-a4d0-cbcc37f7f905_isr_946073_9.stderr
parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2b87ef81e17f9db94f59c4eb6e9ac79 with result from task 876
parsl.dataflow.dflow DEBUG: Task 876 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 5841c1617d072c38b78e2b1b8493f4c0
parsl.dataflow.memoization INFO: Task 877 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 483
parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/be02f030-4f7a-412f-98ce-3441185164d9_characterizeImage_946073_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/be02f030-4f7a-412f-98ce-3441185164d9_characterizeImage_946073_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/bd927200-a795-4d43-8442-25591ffd2c25_isr_946073_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/bd927200-a795-4d43-8442-25591ffd2c25_isr_946073_31.stderr
parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c56345137bfba33522ea6a4b250f4f52 with result from task 1104
parsl.dataflow.dflow DEBUG: Task 1104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 03bc3e38ba82b253452d0e68f26bd91a
parsl.dataflow.memoization INFO: Task 1105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 484
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/20d39a7a-72d4-42fb-b170-0af7f3c398bf_characterizeImage_946182_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/20d39a7a-72d4-42fb-b170-0af7f3c398bf_characterizeImage_946182_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f454fbba-5ebb-4059-b0a0-64a4b0383ce4_isr_946182_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f454fbba-5ebb-4059-b0a0-64a4b0383ce4_isr_946182_5.stderr
parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 632be50013ee99d84fea462ea0b02e5b with result from task 873
parsl.dataflow.dflow DEBUG: Task 873 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 874 has memoization hash fefc161d0551720011b1350ec7d15a4e
parsl.dataflow.memoization INFO: Task 874 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 485
parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/108e573b-2521-4cd2-9fd9-8ec299c5b125_characterizeImage_946181_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/108e573b-2521-4cd2-9fd9-8ec299c5b125_characterizeImage_946181_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/823b260f-2599-475a-aafb-72f29b17237b_isr_946181_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/823b260f-2599-475a-aafb-72f29b17237b_isr_946181_22.stderr
parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7654fafb8934e20e6a40b68f1b17728f with result from task 411
parsl.dataflow.dflow DEBUG: Task 411 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 4aca5057043992708ab01972a0a4bb25
parsl.dataflow.memoization INFO: Task 412 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 486
parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/492a7a9b-9283-4cbb-9e16-e071bc0ab497_characterizeImage_946072_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/492a7a9b-9283-4cbb-9e16-e071bc0ab497_characterizeImage_946072_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/67a799da-bef2-4a35-a889-8093ee9da949_isr_946072_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/67a799da-bef2-4a35-a889-8093ee9da949_isr_946072_47.stderr
parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7acb6bba393cc4dc3af637db83b19fc1 with result from task 882
parsl.dataflow.dflow DEBUG: Task 882 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 883 has memoization hash ab9af1621c67ab41cf4e4fae5381019e
parsl.dataflow.memoization INFO: Task 883 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 487
parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/df3164b7-0643-4cba-9027-7bedaa231722_characterizeImage_946180_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/df3164b7-0643-4cba-9027-7bedaa231722_characterizeImage_946180_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f4b22f69-bed1-4515-b682-8566aeeb9d90_isr_946180_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/f4b22f69-bed1-4515-b682-8566aeeb9d90_isr_946180_49.stderr
parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86640b50e4f4c4c2f5160320f52a421c with result from task 1041
parsl.dataflow.dflow DEBUG: Task 1041 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 24c000b37834fe74b0912fa15a1e5864
parsl.dataflow.memoization INFO: Task 1042 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 488
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d065d8b4-fd29-439f-ac78-c9c5db1b0f1f_characterizeImage_946072_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d065d8b4-fd29-439f-ac78-c9c5db1b0f1f_characterizeImage_946072_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d6fdf13f-b70c-4704-8c4e-fd1101938f36_isr_946072_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d6fdf13f-b70c-4704-8c4e-fd1101938f36_isr_946072_20.stderr
parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfbe64c69bf198d85c2c1bdb951d3db5 with result from task 393
parsl.dataflow.dflow DEBUG: Task 393 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 5e78e5dfb34761251b6a28ed9ee6b65b
parsl.dataflow.memoization INFO: Task 394 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 489
parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/86cf2203-d675-471e-8f19-3135dbad3234_characterizeImage_946181_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/86cf2203-d675-471e-8f19-3135dbad3234_characterizeImage_946181_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/30296992-c294-4dd6-a2ef-eed532683c3e_isr_946181_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/30296992-c294-4dd6-a2ef-eed532683c3e_isr_946181_37.stderr
parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04f6a236dbc3c1c3c8d4d6b62d9f0219 with result from task 759
parsl.dataflow.dflow DEBUG: Task 759 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 3681e3b80e11ae04afcda1596f51d633
parsl.dataflow.memoization INFO: Task 760 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 490
parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c23519c8-f300-428a-9dab-16a773a4f790_characterizeImage_946182_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c23519c8-f300-428a-9dab-16a773a4f790_characterizeImage_946182_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/bc0ee1a7-576d-4f35-b49b-694a1fc5fbce_isr_946182_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/bc0ee1a7-576d-4f35-b49b-694a1fc5fbce_isr_946182_52.stderr
parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9998b46f363aad361bb789a0e5d2ca0 with result from task 726
parsl.dataflow.dflow DEBUG: Task 726 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 727 has memoization hash fc982968638e22f813bc859177322aac
parsl.dataflow.memoization INFO: Task 727 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 491
parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ce612a0a-68a1-4201-b58b-57cd14f21363_characterizeImage_946180_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ce612a0a-68a1-4201-b58b-57cd14f21363_characterizeImage_946180_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2539d1b2-734f-4a51-ab2f-6c07f81422f2_isr_946180_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2539d1b2-734f-4a51-ab2f-6c07f81422f2_isr_946180_36.stderr
parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81324246537705f5e01cf1d033d696de with result from task 813
parsl.dataflow.dflow DEBUG: Task 813 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 814 has memoization hash b77741570436052dbe01bf0554ffd820
parsl.dataflow.memoization INFO: Task 814 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 492
parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5768c43c-dbbf-4115-a4e3-f382449fe3c1_characterizeImage_946074_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5768c43c-dbbf-4115-a4e3-f382449fe3c1_characterizeImage_946074_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fe86de84-b263-468c-8628-16103509a959_isr_946074_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fe86de84-b263-468c-8628-16103509a959_isr_946074_35.stderr
parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 08396f2a2e5e6782bae39ccf9979455f with result from task 717
parsl.dataflow.dflow DEBUG: Task 717 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 51a94bb611e4d90390ac7631cc979e97
parsl.dataflow.memoization INFO: Task 718 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 493
parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/091e1f30-74de-45f9-aea1-a2c1e37b1e9d_characterizeImage_946072_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/091e1f30-74de-45f9-aea1-a2c1e37b1e9d_characterizeImage_946072_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4df5c816-e20b-4e8d-8651-be1a740398d9_isr_946072_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4df5c816-e20b-4e8d-8651-be1a740398d9_isr_946072_62.stderr
parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e556b8d469fa3fc95edb03801b8a9f37 with result from task 696
parsl.dataflow.dflow DEBUG: Task 696 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 697 has memoization hash d04e4c23ad0f8f57312b392aff1eb89f
parsl.dataflow.memoization INFO: Task 697 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 494
parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fddf67d9-c763-40c7-a76e-ade24e71f0da_characterizeImage_946182_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fddf67d9-c763-40c7-a76e-ade24e71f0da_characterizeImage_946182_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3daa38cd-cd51-4c14-a606-79aaa3186454_isr_946182_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3daa38cd-cd51-4c14-a606-79aaa3186454_isr_946182_20.stderr
parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e1865cc1fdfac1ba88efc01aaeb1b2f with result from task 1098
parsl.dataflow.dflow DEBUG: Task 1098 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash a6de56ef2acd9bfc36c8f7da7e9d4366
parsl.dataflow.memoization INFO: Task 1099 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 495
parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/64b53b62-ca49-4924-89b3-3736b180beca_characterizeImage_946073_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/64b53b62-ca49-4924-89b3-3736b180beca_characterizeImage_946073_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e8b4c20-0116-4b44-a86c-d4b5cdd575b2_isr_946073_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/4e8b4c20-0116-4b44-a86c-d4b5cdd575b2_isr_946073_4.stderr
parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2769e650d5a982f6153d37ac39e72be9 with result from task 282
parsl.dataflow.dflow DEBUG: Task 282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 283 has memoization hash c0c89228bcc3126a9efe85f27f974d35
parsl.dataflow.memoization INFO: Task 283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 496
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/adbe60e2-97b7-4276-8594-cd65e46a649e_characterizeImage_946072_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/adbe60e2-97b7-4276-8594-cd65e46a649e_characterizeImage_946072_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c1a92695-b2b4-481b-becd-e05e2d30b91b_isr_946072_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c1a92695-b2b4-481b-becd-e05e2d30b91b_isr_946072_8.stderr
parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9e5e75265fc0bfe8607a1f5e79f538e with result from task 279
parsl.dataflow.dflow DEBUG: Task 279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 691b7287b497dd2c55af1ab2063b88ad
parsl.dataflow.memoization INFO: Task 280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 497
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/de40cdd6-2497-4a56-83b9-ac194235c466_characterizeImage_946072_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/de40cdd6-2497-4a56-83b9-ac194235c466_characterizeImage_946072_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fa95802-4f58-46b9-96af-3ae47ea2fc33_isr_946072_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fa95802-4f58-46b9-96af-3ae47ea2fc33_isr_946072_16.stderr
parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db3de542c3c00966432d3dbee30137b5 with result from task 261
parsl.dataflow.dflow DEBUG: Task 261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 262 has memoization hash d44f95cd04c00c186953d6714287b9cd
parsl.dataflow.memoization INFO: Task 262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 498
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9996e73d-2cf8-4cce-92c9-81e867cb3a66_characterizeImage_946074_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9996e73d-2cf8-4cce-92c9-81e867cb3a66_characterizeImage_946074_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d8dd6500-509d-41ce-9253-57491381d36f_isr_946074_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/d8dd6500-509d-41ce-9253-57491381d36f_isr_946074_38.stderr
parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 449e1b089351e8a67dbbc56d2a2e411d with result from task 273
parsl.dataflow.dflow DEBUG: Task 273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 612cba8d44b5f75241669d638631c5ca
parsl.dataflow.memoization INFO: Task 274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 499
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/803bc082-9adb-4b3b-9432-e916231308bb_characterizeImage_946073_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/803bc082-9adb-4b3b-9432-e916231308bb_characterizeImage_946073_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9d0cbb84-14ef-44a3-81cb-a8ef1d21ecd6_isr_946073_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9d0cbb84-14ef-44a3-81cb-a8ef1d21ecd6_isr_946073_1.stderr
parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b58634714295803df128c429dd39172 with result from task 267
parsl.dataflow.dflow DEBUG: Task 267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 667ab15f04f1ba456465b3436d56caa3
parsl.dataflow.memoization INFO: Task 268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 500
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4c593004-ef7b-4aa2-b26e-e2b03f8df216_characterizeImage_946073_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4c593004-ef7b-4aa2-b26e-e2b03f8df216_characterizeImage_946073_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8e36f6a3-0659-4fc5-b2b7-b3caafc7fbc8_isr_946073_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8e36f6a3-0659-4fc5-b2b7-b3caafc7fbc8_isr_946073_5.stderr
parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24fe9c4215def8d5afb5c00663c4f856 with result from task 237
parsl.dataflow.dflow DEBUG: Task 237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 6f827f6e6593b8aa095d18e61c74f21f
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 501
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e016afa8-63a3-4036-a393-075938c4f72d_characterizeImage_946181_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e016afa8-63a3-4036-a393-075938c4f72d_characterizeImage_946181_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c4b61661-4e98-4f84-b0dd-57ce27a1cd99_isr_946181_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c4b61661-4e98-4f84-b0dd-57ce27a1cd99_isr_946181_25.stderr
parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5df2557b78f285bfd6849eb1282065b with result from task 255
parsl.dataflow.dflow DEBUG: Task 255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 033da33d28c34395ddc1b62a469960ad
parsl.dataflow.memoization INFO: Task 256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 502
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ab4fe870-2d2a-49f6-a9bb-59e02d216ae2_characterizeImage_946180_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ab4fe870-2d2a-49f6-a9bb-59e02d216ae2_characterizeImage_946180_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ee920608-b070-4cee-adc3-3be63698ca62_isr_946180_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ee920608-b070-4cee-adc3-3be63698ca62_isr_946180_51.stderr
parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bbbaa58988e6ea28ccd7793708f5ec08 with result from task 258
parsl.dataflow.dflow DEBUG: Task 258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 259 has memoization hash dac2f60e20db33d7e3c1287b64c865d6
parsl.dataflow.memoization INFO: Task 259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 503
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/810b7c05-e801-436e-8015-d4e499d71ecc_characterizeImage_946182_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/810b7c05-e801-436e-8015-d4e499d71ecc_characterizeImage_946182_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8c806c12-92c2-40ee-a082-775969ac827e_isr_946182_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/8c806c12-92c2-40ee-a082-775969ac827e_isr_946182_30.stderr
parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac199d6f872ea36b507fe766ae107039 with result from task 252
parsl.dataflow.dflow DEBUG: Task 252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 41706e42b16fab43250ccfb9480de2b0
parsl.dataflow.memoization INFO: Task 253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 504
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c9a78ef7-086c-46b2-8ce5-6830cb310492_characterizeImage_946180_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c9a78ef7-086c-46b2-8ce5-6830cb310492_characterizeImage_946180_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/85d53ed5-7a47-4b23-8bff-8984d4fb20d2_isr_946180_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/85d53ed5-7a47-4b23-8bff-8984d4fb20d2_isr_946180_33.stderr
parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb03623f3500a0d1b5541797f5dd8e85 with result from task 240
parsl.dataflow.dflow DEBUG: Task 240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 88912704e07d7ff20f00531476595ef0
parsl.dataflow.memoization INFO: Task 241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 505
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ec9c181b-6462-45ac-81f3-afbc5806fa36_characterizeImage_946182_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ec9c181b-6462-45ac-81f3-afbc5806fa36_characterizeImage_946182_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/81c3c77b-1ca0-491a-a427-00e95949a107_isr_946182_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/81c3c77b-1ca0-491a-a427-00e95949a107_isr_946182_11.stderr
parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3bde39877181e10664dcc1339a290554 with result from task 243
parsl.dataflow.dflow DEBUG: Task 243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 3d8a763c3d6abdb297635f46daac598f
parsl.dataflow.memoization INFO: Task 244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 506
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/71b3c84d-2bae-4b84-be72-aab9af3ed4dd_characterizeImage_946072_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/71b3c84d-2bae-4b84-be72-aab9af3ed4dd_characterizeImage_946072_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/436948bb-cbdf-4b46-befb-411f4817efbe_isr_946072_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/436948bb-cbdf-4b46-befb-411f4817efbe_isr_946072_14.stderr
parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0127e05fb1836e9e35b50d08e4ac828b with result from task 246
parsl.dataflow.dflow DEBUG: Task 246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 247 has memoization hash ceddc80ff1c779f966a0426d4c516bea
parsl.dataflow.memoization INFO: Task 247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 507
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/acb69f8c-9e0d-49f0-b6e1-45c7b5acb8bf_characterizeImage_946072_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/acb69f8c-9e0d-49f0-b6e1-45c7b5acb8bf_characterizeImage_946072_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/deb10df4-76e0-4255-a340-2e28e519cf20_isr_946072_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/deb10df4-76e0-4255-a340-2e28e519cf20_isr_946072_28.stderr
parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb937feed5176a5f97ac350e183046af with result from task 249
parsl.dataflow.dflow DEBUG: Task 249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 4ae6c3d0b03d8ca1fe672cfc7570ea91
parsl.dataflow.memoization INFO: Task 250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 508
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/291c352b-9f81-490a-a7d9-36ecbe428315_characterizeImage_946073_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/291c352b-9f81-490a-a7d9-36ecbe428315_characterizeImage_946073_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/19a32372-bf00-4ca4-a27a-54498b3fe4dc_isr_946073_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/19a32372-bf00-4ca4-a27a-54498b3fe4dc_isr_946073_16.stderr
parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f881c3fd8f191e2c42fdd151ffd5f1f8 with result from task 264
parsl.dataflow.dflow DEBUG: Task 264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 9aeed152a2363089a08e81f07922f45d
parsl.dataflow.memoization INFO: Task 265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 509
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/aa43edf6-8822-44d4-a47c-ac42d8982b45_characterizeImage_946074_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/aa43edf6-8822-44d4-a47c-ac42d8982b45_characterizeImage_946074_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/84d3eabc-b73e-47ef-99d7-d0b74e2f927c_isr_946074_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/84d3eabc-b73e-47ef-99d7-d0b74e2f927c_isr_946074_6.stderr
parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5de9e88ccc7a7c28ea1d0f31c45fc9ec with result from task 276
parsl.dataflow.dflow DEBUG: Task 276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 97f6d2da9b8873bee3668b4b6a985faa
parsl.dataflow.memoization INFO: Task 277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 510
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/97d29b98-683c-4966-9f83-240aac503d98_characterizeImage_946073_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/97d29b98-683c-4966-9f83-240aac503d98_characterizeImage_946073_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b2aee601-7495-4d94-a497-a5241957fd51_isr_946073_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b2aee601-7495-4d94-a497-a5241957fd51_isr_946073_49.stderr
parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d74af8dc33f61788ecfb7d012c6740ed with result from task 270
parsl.dataflow.dflow DEBUG: Task 270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 7289cc915d8bfb42d88b1b025c8b64aa
parsl.dataflow.memoization INFO: Task 271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 511
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/52c2ea5c-ac0a-4eb0-bc8d-f6d67cfb877b_characterizeImage_946180_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/52c2ea5c-ac0a-4eb0-bc8d-f6d67cfb877b_characterizeImage_946180_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e666065e-992e-40eb-99cd-238496412a85_isr_946180_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e666065e-992e-40eb-99cd-238496412a85_isr_946180_5.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba0c2d02953fec878d959ca11dcea4a6 with result from task 705
parsl.dataflow.dflow DEBUG: Task 705 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 643211f7ad10ad1e7eefc5dfda02300d
parsl.dataflow.memoization INFO: Task 706 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 512
parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de322350-1597-4a53-b0be-055098845296_characterizeImage_946074_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de322350-1597-4a53-b0be-055098845296_characterizeImage_946074_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c5f11ee4-cc6c-462f-9a6b-9fea5d78bb69_isr_946074_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c5f11ee4-cc6c-462f-9a6b-9fea5d78bb69_isr_946074_34.stderr
parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0535647ea9c3fad4179a5103e56884 with result from task 693
parsl.dataflow.dflow DEBUG: Task 693 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 8c32611fdd5c7dc965eab47de194ebee
parsl.dataflow.memoization INFO: Task 694 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 513
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0b7f3802-88c5-4450-85b4-45a68ae7d823_characterizeImage_946072_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0b7f3802-88c5-4450-85b4-45a68ae7d823_characterizeImage_946072_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/5e0a1768-9a78-41d3-997f-b66afd470e53_isr_946072_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/5e0a1768-9a78-41d3-997f-b66afd470e53_isr_946072_9.stderr
parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7202cc674f520cebffb13479b97804d with result from task 702
parsl.dataflow.dflow DEBUG: Task 702 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 09504d597b158032f7171d5a2325ddda
parsl.dataflow.memoization INFO: Task 703 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 514
parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/95aaf9e1-9584-4962-a29d-282677bca3f9_characterizeImage_946182_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/95aaf9e1-9584-4962-a29d-282677bca3f9_characterizeImage_946182_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/01aafa68-81fa-4c6d-a827-ba9743a461a9_isr_946182_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/01aafa68-81fa-4c6d-a827-ba9743a461a9_isr_946182_57.stderr
parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67548aad3e6901d89c09b292a746849e with result from task 687
parsl.dataflow.dflow DEBUG: Task 687 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 688 has memoization hash b389b039cfeb16b0e1fa5d59156c95b5
parsl.dataflow.memoization INFO: Task 688 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 515
parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/bc018c03-a417-4862-b532-1c7e8f2ec9d5_characterizeImage_946180_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/bc018c03-a417-4862-b532-1c7e8f2ec9d5_characterizeImage_946180_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e6bae503-d94a-4baa-aecc-cc14e43c976f_isr_946180_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e6bae503-d94a-4baa-aecc-cc14e43c976f_isr_946180_26.stderr
parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1bcf9ec4d443b949b2c496c3288828cd with result from task 708
parsl.dataflow.dflow DEBUG: Task 708 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 2bf5ebcfe6021e8664cdbaa146f153c0
parsl.dataflow.memoization INFO: Task 709 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 516
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/542e2ab1-3a6a-4a41-899a-da680986d68e_characterizeImage_946180_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/542e2ab1-3a6a-4a41-899a-da680986d68e_characterizeImage_946180_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9c7ce740-8655-47dd-8d57-a2be38fb8624_isr_946180_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/9c7ce740-8655-47dd-8d57-a2be38fb8624_isr_946180_4.stderr
parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28301de5ba44fafb42b4dea6cb814f0c with result from task 699
parsl.dataflow.dflow DEBUG: Task 699 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 700 has memoization hash f9a3ac9e2e723f8adbdc116f3ca828c4
parsl.dataflow.memoization INFO: Task 700 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 517
parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfa663d7-eb8c-40f2-b0d7-e27163075121_characterizeImage_946182_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfa663d7-eb8c-40f2-b0d7-e27163075121_characterizeImage_946182_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2417af8f-e939-45d0-8cce-8fe9cd269686_isr_946182_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2417af8f-e939-45d0-8cce-8fe9cd269686_isr_946182_6.stderr
parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dede1e74e2072beb53e6062619c3f880 with result from task 690
parsl.dataflow.dflow DEBUG: Task 690 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 63b0f17c59cbd1c8629cbe92936f6248
parsl.dataflow.memoization INFO: Task 691 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 518
parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a87b8a5c-69be-4866-94e7-6620c27fb126_characterizeImage_946073_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a87b8a5c-69be-4866-94e7-6620c27fb126_characterizeImage_946073_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/224a8166-0ac3-4b97-b057-8c3316fb0038_isr_946073_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/224a8166-0ac3-4b97-b057-8c3316fb0038_isr_946073_59.stderr
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 430 try 0 failed
parsl.dataflow.dflow INFO: Task 430 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 574cab65289e73b66f21d667fed0a3e9
parsl.dataflow.memoization INFO: Task 430 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 430 try 1 launched on executor multi with executor id 519
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:49 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:50 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:51 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:51 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:51 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:51 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:51 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:51 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:52 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:53 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:53 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:53 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:53 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:53 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:53 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:54 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:54 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:54 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:54 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:54 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81916b39c6c1de45a14b93c1404859af with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 7e26c6b37543ccafcace72db1e2a43b4
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 520
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/980d2f69-d781-4e6a-b1fe-9fc97a0693ba_characterizeImage_946180_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/980d2f69-d781-4e6a-b1fe-9fc97a0693ba_characterizeImage_946180_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c0cdfdd6-0a35-4a7f-87f5-7303c702a3f1_isr_946180_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c0cdfdd6-0a35-4a7f-87f5-7303c702a3f1_isr_946180_7.stderr
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac6c3a5731b4deb0b00a6baae6a97b89 with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 7236e29880a18f7a32c964d347c11829
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 521
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/94a3a18a-6cfc-493e-b8e6-42aba2e2dd4f_characterizeImage_946074_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/94a3a18a-6cfc-493e-b8e6-42aba2e2dd4f_characterizeImage_946074_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/36aee1d1-7c58-4e1f-b922-da2934361737_isr_946074_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/36aee1d1-7c58-4e1f-b922-da2934361737_isr_946074_7.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 926c54112ec1c9f0dac305a734055796 with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 02c26b40605cc6830891f42879119f9f
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 522
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/19fbbfca-f9b8-4dfa-aa70-5e452bae49b8_characterizeImage_946072_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/19fbbfca-f9b8-4dfa-aa70-5e452bae49b8_characterizeImage_946072_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8d2804a7-3fd2-4b7c-964d-d55bfb603451_isr_946072_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8d2804a7-3fd2-4b7c-964d-d55bfb603451_isr_946072_50.stderr
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4366b692210121ee387dad75f39fcd2c with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash a61dab9813e6b5cd16359090bd2dd131
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 523
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bb24e41a-813e-46f1-a466-cc104fb0a26f_characterizeImage_946182_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bb24e41a-813e-46f1-a466-cc104fb0a26f_characterizeImage_946182_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/68f8d5ce-3270-4af1-ad99-8011b4db20d3_isr_946182_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/68f8d5ce-3270-4af1-ad99-8011b4db20d3_isr_946182_19.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5dd95028311b77f3a46714c178c911 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 920a28fbf4bb82ed7c80c75a10a55e58
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 524
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ac29bff-c448-4ac9-a166-a4c31beb2b3b_characterizeImage_946074_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ac29bff-c448-4ac9-a166-a4c31beb2b3b_characterizeImage_946074_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8dbb4db9-035f-4d90-9692-20b384b28b7e_isr_946074_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8dbb4db9-035f-4d90-9692-20b384b28b7e_isr_946074_37.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ceaf182d379b70fc9b13faa409c5ff0c with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 9224c5eb454c698536b03094e9457056
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 525
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dbbde51f-57c4-4f0d-a6eb-d67da10e9e67_characterizeImage_946074_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dbbde51f-57c4-4f0d-a6eb-d67da10e9e67_characterizeImage_946074_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/37f5f22c-6613-442a-b62f-49049d83d713_isr_946074_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/37f5f22c-6613-442a-b62f-49049d83d713_isr_946074_10.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 554fd98c41f58d713222474fa7f9fb9f with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 655dd6e68ebb0f0be9e406e08696e5ec
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 526
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1f44b62d-2103-4839-906a-81c2284473ff_characterizeImage_946072_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1f44b62d-2103-4839-906a-81c2284473ff_characterizeImage_946072_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/af2c0791-f400-4ace-bf51-8dbd0b72898e_isr_946072_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/af2c0791-f400-4ace-bf51-8dbd0b72898e_isr_946072_44.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 384eab375465ed547f418045adef00d5 with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 67b33bfe3f2b0819053d0c6cb8cb414a
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 527
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/41eabc99-0b4d-4672-8589-f6df5c705122_characterizeImage_946182_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/41eabc99-0b4d-4672-8589-f6df5c705122_characterizeImage_946182_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f466c88e-5d4e-488b-8802-a69c2be083a5_isr_946182_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/f466c88e-5d4e-488b-8802-a69c2be083a5_isr_946182_44.stderr
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d8c7107994466e70d7896ccdf68ecc9 with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 341701b9a22f32d158764d3771495f95
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 528
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8f224ac8-7b26-45ef-a478-f51b318fb7ff_characterizeImage_946180_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8f224ac8-7b26-45ef-a478-f51b318fb7ff_characterizeImage_946180_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d069d79b-93f7-417d-ac26-6044ab198beb_isr_946180_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d069d79b-93f7-417d-ac26-6044ab198beb_isr_946180_60.stderr
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee3157a937ee5bad04add3061cbb1cab with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 1b01803ea529e8f891eaada3e4399e5e
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f884a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 529
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d1eeb0db-1c51-46ec-9437-3081642a579a_characterizeImage_946073_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d1eeb0db-1c51-46ec-9437-3081642a579a_characterizeImage_946073_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ba443340-e3da-4ede-a5c8-4625efe00941_isr_946073_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ba443340-e3da-4ede-a5c8-4625efe00941_isr_946073_6.stderr
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4ec301342e1d742f55c8187087d839 with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash eeed018bae1e716cd203f418e0c8fbcd
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 530
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6628ce72-89bd-477e-bc87-3d3889c16a30_characterizeImage_946072_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6628ce72-89bd-477e-bc87-3d3889c16a30_characterizeImage_946072_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2739ffb6-a372-4193-b835-337e4e57c37d_isr_946072_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2739ffb6-a372-4193-b835-337e4e57c37d_isr_946072_11.stderr
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd647351e9efd02ebd9c49cf594ce7d6 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash bd8953c8d801f05e7d6dcf6abff8b4ad
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:54 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 531
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b995bb2-e188-431a-ae07-1aa99731236b_characterizeImage_946073_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b995bb2-e188-431a-ae07-1aa99731236b_characterizeImage_946073_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e1768d48-e88a-4b05-81a5-7b0b37d5d7a2_isr_946073_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/e1768d48-e88a-4b05-81a5-7b0b37d5d7a2_isr_946073_39.stderr
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5f3d9f4046c6d0d0471d1a9ebc2daa with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash f579f5d431bb4714c67907a557e522fa
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 532
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7acd4-0e1b-4d21-a784-1d05d2c310ba_characterizeImage_946073_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7acd4-0e1b-4d21-a784-1d05d2c310ba_characterizeImage_946073_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/82938aa3-03d1-4a90-a7a0-5a3bbd25bbe4_isr_946073_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/82938aa3-03d1-4a90-a7a0-5a3bbd25bbe4_isr_946073_23.stderr
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3ff3ce0a99ae383275ef37244d2e3b80 with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash e9b3725bc86aa471b99334c57e5bbe82
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 533
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ae255fb8-97c9-4de5-a4b1-72cd37f93775_characterizeImage_946074_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ae255fb8-97c9-4de5-a4b1-72cd37f93775_characterizeImage_946074_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5c0a64a9-8a0a-43bc-a1d9-0bcd8ac0820a_isr_946074_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5c0a64a9-8a0a-43bc-a1d9-0bcd8ac0820a_isr_946074_20.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 042869ada14de0386d9480d7f5126763 with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash d943c01ff235fbd9f4fb653671fb7b11
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 534
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/62e8d991-7177-4627-8304-930d48b3b945_characterizeImage_946181_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/62e8d991-7177-4627-8304-930d48b3b945_characterizeImage_946181_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b382c1d0-bb7e-4f35-afa1-6bf74e3a0694_isr_946181_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b382c1d0-bb7e-4f35-afa1-6bf74e3a0694_isr_946181_41.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0603a519fa10c85115ad9b47a66c9db with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 61ce131e92ea8e8bde43c1264bd56d7e
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f887c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 535
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/908dcb9e-d306-402b-a708-6f4f8e1715e0_characterizeImage_946180_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/908dcb9e-d306-402b-a708-6f4f8e1715e0_characterizeImage_946180_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/07a677cb-555b-46f8-bb0e-c8dcbbda989d_isr_946180_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/07a677cb-555b-46f8-bb0e-c8dcbbda989d_isr_946180_30.stderr
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67b8d71ad99cf1e705bec2b967b59928 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 7d88122435be93180e125a38d6da7f2f
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 536
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cf003b05-e86e-45c7-adfb-cad778db261f_characterizeImage_946072_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cf003b05-e86e-45c7-adfb-cad778db261f_characterizeImage_946072_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4ee107b8-0be2-4a74-8b4a-7d3edd0b1454_isr_946072_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4ee107b8-0be2-4a74-8b4a-7d3edd0b1454_isr_946072_60.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f1a8a468f5c1b4da86aca6d0ce591102 with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 34d8d43cb9d49c433c89d9d945f4d4d2
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 537
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e32c4416-778d-4a05-8851-086d17232d1f_characterizeImage_946180_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e32c4416-778d-4a05-8851-086d17232d1f_characterizeImage_946180_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/14e1a92c-7ea6-4704-ba9b-5922941867f6_isr_946180_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/14e1a92c-7ea6-4704-ba9b-5922941867f6_isr_946180_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95da361f14be3f91158b4b695d1ea5b9 with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash d8a250875b313b1f8b0540938dad4d93
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 538
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5fc3035b-f3ed-41e2-be7a-61526e418679_characterizeImage_946180_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5fc3035b-f3ed-41e2-be7a-61526e418679_characterizeImage_946180_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/8ded0a21-2aa0-4c4e-a260-acc8298885a7_isr_946180_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/8ded0a21-2aa0-4c4e-a260-acc8298885a7_isr_946180_24.stderr
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22869fe627e93bedc97c2cf0c64c08cf with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 3b0e6abff0b2e74244f35ef4ec944995
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 539
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/3578c3de-c1b0-46e9-9138-6f8c4a8c90a1_characterizeImage_946072_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/3578c3de-c1b0-46e9-9138-6f8c4a8c90a1_characterizeImage_946072_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d1c13ab8-f554-4301-9a1f-fbd98aeb7823_isr_946072_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d1c13ab8-f554-4301-9a1f-fbd98aeb7823_isr_946072_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:55 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:56 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:57 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:58 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:35:59 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:00 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:01 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:01 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:01 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:01 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:01 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 895a662cc9ba6a597a1bd8604dc6a498 with result from task 630
parsl.dataflow.dflow DEBUG: Task 630 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 631 has memoization hash c184be597c4d641c47a6d793ca418556
parsl.dataflow.memoization INFO: Task 631 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 540
parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4eeea007-f5fa-430d-ae82-52bd3f281d54_characterizeImage_946072_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4eeea007-f5fa-430d-ae82-52bd3f281d54_characterizeImage_946072_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8ce900f6-829c-465d-b474-1ac34ae6098e_isr_946072_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8ce900f6-829c-465d-b474-1ac34ae6098e_isr_946072_32.stderr
parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16dfd65255226c6d6fb231e0fae67032 with result from task 618
parsl.dataflow.dflow DEBUG: Task 618 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 251c79f8bef17953c3a4a1b48e466225
parsl.dataflow.memoization INFO: Task 619 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088edbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 541
parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/75a864b3-60f0-4501-8883-26fad1e48bd6_characterizeImage_946182_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/75a864b3-60f0-4501-8883-26fad1e48bd6_characterizeImage_946182_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6b8ab4b1-64f3-48da-a327-a1a1e0eb059c_isr_946182_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/6b8ab4b1-64f3-48da-a327-a1a1e0eb059c_isr_946182_8.stderr
parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31c61a2b8d90f93c435f99dc834489e2 with result from task 636
parsl.dataflow.dflow DEBUG: Task 636 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 560fb2eff08dae1b6dae504304f2cf17
parsl.dataflow.memoization INFO: Task 637 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 542
parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9591292d-54a7-4efa-a76e-9f70dfd11824_characterizeImage_946073_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9591292d-54a7-4efa-a76e-9f70dfd11824_characterizeImage_946073_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/da95c5f4-c81a-4894-bcbd-6549bbaa07c7_isr_946073_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/da95c5f4-c81a-4894-bcbd-6549bbaa07c7_isr_946073_32.stderr
parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6bd265e76daad9ddd5071103ef750b00 with result from task 621
parsl.dataflow.dflow DEBUG: Task 621 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 835595608ca5788686792b6315eaf39d
parsl.dataflow.memoization INFO: Task 622 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088edee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 543
parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/31240a4e-ec49-4f3b-a539-a56472d32608_characterizeImage_946072_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/31240a4e-ec49-4f3b-a539-a56472d32608_characterizeImage_946072_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/aaae5965-a612-4dd8-8f81-daa8a835b5a0_isr_946072_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/aaae5965-a612-4dd8-8f81-daa8a835b5a0_isr_946072_5.stderr
parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0efeb25e915a0e789f636752b88fab13 with result from task 627
parsl.dataflow.dflow DEBUG: Task 627 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 5fdc61910e4b890d260bbd250b224b90
parsl.dataflow.memoization INFO: Task 628 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 544
parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/683c43ba-40a0-4835-bf9a-d79183ef8f01_characterizeImage_946181_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/683c43ba-40a0-4835-bf9a-d79183ef8f01_characterizeImage_946181_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/68b2953c-da29-4d18-854a-a9b534772bb8_isr_946181_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/68b2953c-da29-4d18-854a-a9b534772bb8_isr_946181_39.stderr
parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bbce37f623b23accf9563c03a791d941 with result from task 633
parsl.dataflow.dflow DEBUG: Task 633 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 8a99afa4eb9ed6ff4f0daaa813b24a03
parsl.dataflow.memoization INFO: Task 634 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 545
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a336202d-ae2f-4ec9-9a68-d43dc46a262a_isr_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/a336202d-ae2f-4ec9-9a68-d43dc46a262a_isr_946074_2.stderr
parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d23bb6260d4784799e3512038c2c31a5 with result from task 615
parsl.dataflow.dflow DEBUG: Task 615 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 616 has memoization hash cc934904e8211c9beb8ca6aa3e2bef2a
parsl.dataflow.memoization INFO: Task 616 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 546
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7d42fd1b-dd37-4ffe-a5a5-89bd6dd9eead_characterizeImage_946073_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7d42fd1b-dd37-4ffe-a5a5-89bd6dd9eead_characterizeImage_946073_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9a7d1483-ae0b-422a-9430-f5a9de782b05_isr_946073_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9a7d1483-ae0b-422a-9430-f5a9de782b05_isr_946073_47.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21c4fb80d598b47efc545881b157837f with result from task 624
parsl.dataflow.dflow DEBUG: Task 624 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 5dfc53ed9c7fa8230978592b0fcd8055
parsl.dataflow.memoization INFO: Task 625 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 547
parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d6c29223-5ccd-4cb9-b848-de00f17832f1_characterizeImage_946072_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d6c29223-5ccd-4cb9-b848-de00f17832f1_characterizeImage_946072_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/163ceeda-0e88-4f4a-a2af-6289bc324a39_isr_946072_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/163ceeda-0e88-4f4a-a2af-6289bc324a39_isr_946072_36.stderr
parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba645e44ba3171f6bdd3ff6b4d3fd387 with result from task 957
parsl.dataflow.dflow DEBUG: Task 957 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 4654e187a1019bbf2ee9cc60941c0cac
parsl.dataflow.memoization INFO: Task 958 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 548
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2105b570-4538-4d9e-bb38-dd26f5601a7e_characterizeImage_946180_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2105b570-4538-4d9e-bb38-dd26f5601a7e_characterizeImage_946180_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c778f597-5baf-416c-9104-c2bb13cb532c_isr_946180_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c778f597-5baf-416c-9104-c2bb13cb532c_isr_946180_45.stderr
parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70625a6c1e56404905050635c1922341 with result from task 963
parsl.dataflow.dflow DEBUG: Task 963 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 964 has memoization hash d752b1323ca2307d86c02a4cb2e00962
parsl.dataflow.memoization INFO: Task 964 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 549
parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f19878-7089-470e-b04f-d4a41f6cd43b_characterizeImage_946074_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f19878-7089-470e-b04f-d4a41f6cd43b_characterizeImage_946074_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5b067585-4d55-49a3-a30d-e296e149c19d_isr_946074_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/5b067585-4d55-49a3-a30d-e296e149c19d_isr_946074_33.stderr
parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59c34364a6fb2e8cb75e773f927a51c9 with result from task 972
parsl.dataflow.dflow DEBUG: Task 972 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 8edc284970f3d20078185b40589d27eb
parsl.dataflow.memoization INFO: Task 973 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 550
parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/196c7130-ceb9-4e00-bfb1-f64adabeee12_characterizeImage_946072_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/196c7130-ceb9-4e00-bfb1-f64adabeee12_characterizeImage_946072_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/f2ac9769-c7ea-4024-b56a-ce5c2bb4cd91_isr_946072_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/f2ac9769-c7ea-4024-b56a-ce5c2bb4cd91_isr_946072_53.stderr
parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff28a23f99a2a6a1412df5cf289d202e with result from task 933
parsl.dataflow.dflow DEBUG: Task 933 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 934 has memoization hash f593f1cc0fc283882eb0ce9f4eac6e7c
parsl.dataflow.memoization INFO: Task 934 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 551
parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2c6740ef-083b-45d2-a9fb-6d1091b2c059_characterizeImage_946182_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2c6740ef-083b-45d2-a9fb-6d1091b2c059_characterizeImage_946182_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3b4f7912-0c1c-42e3-852c-3ae99a24a784_isr_946182_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/3b4f7912-0c1c-42e3-852c-3ae99a24a784_isr_946182_3.stderr
parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93a878d8437571c2b45c9cb584de7236 with result from task 945
parsl.dataflow.dflow DEBUG: Task 945 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 946 has memoization hash e6ed06ea0117bc08ee79b76465c9895e
parsl.dataflow.memoization INFO: Task 946 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 552
parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/86f3a392-c4eb-42d6-847d-3460291b2f7a_characterizeImage_946074_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/86f3a392-c4eb-42d6-847d-3460291b2f7a_characterizeImage_946074_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9e02ef19-218e-4e82-8a70-3218b3542a7e_isr_946074_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9e02ef19-218e-4e82-8a70-3218b3542a7e_isr_946074_28.stderr
parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c58e68e8994734e8e492c82a9dc0279a with result from task 936
parsl.dataflow.dflow DEBUG: Task 936 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 937 has memoization hash ef440dd7c449b04977e82e079ea770c5
parsl.dataflow.memoization INFO: Task 937 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 553
parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ecec1eec-a978-4110-8d36-89d4ac8bf07d_characterizeImage_946181_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ecec1eec-a978-4110-8d36-89d4ac8bf07d_characterizeImage_946181_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7c434cbb-efd9-4983-9c25-39539998ca79_isr_946181_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7c434cbb-efd9-4983-9c25-39539998ca79_isr_946181_44.stderr
parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32a3e4c358f314282a26876450b6fdfa with result from task 954
parsl.dataflow.dflow DEBUG: Task 954 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 900d97c4d808b0a1a67857ad04d8f873
parsl.dataflow.memoization INFO: Task 955 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 554
parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b40a7168-3924-484e-b771-beadf85db982_characterizeImage_946181_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b40a7168-3924-484e-b771-beadf85db982_characterizeImage_946181_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5715258a-1d8d-4023-9345-7f11d066058c_isr_946181_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5715258a-1d8d-4023-9345-7f11d066058c_isr_946181_54.stderr
parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80d973a7380e0a1690222a6803bf0129 with result from task 942
parsl.dataflow.dflow DEBUG: Task 942 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 79ec4cb058243db3b9faa6afcd78881f
parsl.dataflow.memoization INFO: Task 943 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 555
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de851926-21e1-4789-9e6f-63b089e344b6_characterizeImage_946074_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de851926-21e1-4789-9e6f-63b089e344b6_characterizeImage_946074_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/66307685-3022-4947-a850-df1b47750461_isr_946074_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/66307685-3022-4947-a850-df1b47750461_isr_946074_42.stderr
parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34e2ee4625e13b1712bb3132902af04d with result from task 939
parsl.dataflow.dflow DEBUG: Task 939 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 940 has memoization hash d89d5514b1695a172940d360a6cb6356
parsl.dataflow.memoization INFO: Task 940 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 556
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/4159155a-d314-499d-aa80-34a8ca7c974f_characterizeImage_946180_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/4159155a-d314-499d-aa80-34a8ca7c974f_characterizeImage_946180_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ddc4f1df-c947-41ea-bebe-bbbac5174f28_isr_946180_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ddc4f1df-c947-41ea-bebe-bbbac5174f28_isr_946180_23.stderr
parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a30677552a1ebf51999e29b229f9a812 with result from task 948
parsl.dataflow.dflow DEBUG: Task 948 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 98f7f2936b7bd473484a0f946d8d5422
parsl.dataflow.memoization INFO: Task 949 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 557
parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a24bad96-f3a1-4200-92ca-def46aa70038_characterizeImage_946181_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a24bad96-f3a1-4200-92ca-def46aa70038_characterizeImage_946181_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/4033d9bc-06be-45d6-a1a0-0b618a584ff5_isr_946181_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/4033d9bc-06be-45d6-a1a0-0b618a584ff5_isr_946181_24.stderr
parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0af106549bb499de648150dbf725690c with result from task 951
parsl.dataflow.dflow DEBUG: Task 951 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 952 has memoization hash beb2b6f591023cf56d9ae4ac1b39dc84
parsl.dataflow.memoization INFO: Task 952 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 558
parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/10ee2fdf-17da-4e15-8457-3f3815057faf_characterizeImage_946074_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/10ee2fdf-17da-4e15-8457-3f3815057faf_characterizeImage_946074_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1f88305c-8980-45c6-9ef0-8c37caeb987f_isr_946074_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1f88305c-8980-45c6-9ef0-8c37caeb987f_isr_946074_57.stderr
parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5db1f65d1e2bbba538a8301e45869c1 with result from task 975
parsl.dataflow.dflow DEBUG: Task 975 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 2338b807748c57ad97898eec2ae35522
parsl.dataflow.memoization INFO: Task 976 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 559
parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8b6cc605-e486-4428-976f-0b656fb3f980_characterizeImage_946180_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8b6cc605-e486-4428-976f-0b656fb3f980_characterizeImage_946180_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/12c28eee-2e3a-4bb6-8f66-21cadbbba13f_isr_946180_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/12c28eee-2e3a-4bb6-8f66-21cadbbba13f_isr_946180_58.stderr
parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d69dcfd60063ee2ef5639ed63a0004fa with result from task 960
parsl.dataflow.dflow DEBUG: Task 960 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 978af61145fe565139a3e47dcf758c9f
parsl.dataflow.memoization INFO: Task 961 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 560
parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7f2b2b44-7d9e-4a51-86e8-dd0225cd5f5e_characterizeImage_946074_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7f2b2b44-7d9e-4a51-86e8-dd0225cd5f5e_characterizeImage_946074_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/106fa713-0f3a-4980-b159-ec04d8e302c3_isr_946074_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/106fa713-0f3a-4980-b159-ec04d8e302c3_isr_946074_29.stderr
parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51483a58af9f91fc018b2d511a45008a with result from task 978
parsl.dataflow.dflow DEBUG: Task 978 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 979 has memoization hash ba30454913234955d9cc67b6a7259c77
parsl.dataflow.memoization INFO: Task 979 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 561
parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8161566a-9ef4-4aa5-9a85-9e359ce65b8e_characterizeImage_946074_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8161566a-9ef4-4aa5-9a85-9e359ce65b8e_characterizeImage_946074_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8e53e551-8330-4c43-8783-8546b12754bc_isr_946074_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/8e53e551-8330-4c43-8783-8546b12754bc_isr_946074_43.stderr
parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 503e92cc86d453d2bb43e44751a58fee with result from task 969
parsl.dataflow.dflow DEBUG: Task 969 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 699a5ec418044513609ce686153275be
parsl.dataflow.memoization INFO: Task 970 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 562
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/444ba73c-f344-4bc5-9f63-03c5deb4ef90_characterizeImage_946072_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/444ba73c-f344-4bc5-9f63-03c5deb4ef90_characterizeImage_946072_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6bbc3558-56a0-41e9-a3be-b050662e5f88_isr_946072_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6bbc3558-56a0-41e9-a3be-b050662e5f88_isr_946072_37.stderr
parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68977d43ee69fe05e29414377aa510cc with result from task 966
parsl.dataflow.dflow DEBUG: Task 966 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 84509bd9dad101e1d29e08ba125d0707
parsl.dataflow.memoization INFO: Task 967 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 563
parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e3646bb9-1d4e-49bb-8053-bf3701e1a132_characterizeImage_946073_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e3646bb9-1d4e-49bb-8053-bf3701e1a132_characterizeImage_946073_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/3598b994-7241-4a14-81a7-4fc36b258620_isr_946073_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/3598b994-7241-4a14-81a7-4fc36b258620_isr_946073_22.stderr
parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b598d477588ec0a4bcd125d54126cc5 with result from task 1032
parsl.dataflow.dflow DEBUG: Task 1032 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 785db8a9186ac7d3c0fc5f1fc41535de
parsl.dataflow.memoization INFO: Task 1033 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 564
parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/02077692-e0ad-48e3-911a-a976e8e566aa_characterizeImage_946181_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/02077692-e0ad-48e3-911a-a976e8e566aa_characterizeImage_946181_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/fb419264-3050-43fe-aca8-6ab5b8350755_isr_946181_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/fb419264-3050-43fe-aca8-6ab5b8350755_isr_946181_15.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7593d2dc93d1552323edfb93ccbe754c with result from task 1035
parsl.dataflow.dflow DEBUG: Task 1035 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash e66cc382658768d6483bf4cf09119aa7
parsl.dataflow.memoization INFO: Task 1036 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 565
parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/855ae4f9-aba5-454e-a8be-d74bfcd70c42_characterizeImage_946182_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/855ae4f9-aba5-454e-a8be-d74bfcd70c42_characterizeImage_946182_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/28cf9bba-2443-48ca-b20a-c45c4b4fb92f_isr_946182_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/28cf9bba-2443-48ca-b20a-c45c4b4fb92f_isr_946182_29.stderr
parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac9a6659535123dbee0a3d69f6fa401 with result from task 570
parsl.dataflow.dflow DEBUG: Task 570 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 1a4a89d82befa5605b68a4c99c253420
parsl.dataflow.memoization INFO: Task 571 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 566
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fd4a72a3-145f-4647-ad61-13e60a50d3d9_characterizeImage_946074_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fd4a72a3-145f-4647-ad61-13e60a50d3d9_characterizeImage_946074_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4c1c8608-0eb8-402e-b52a-7569f9e66aa2_isr_946074_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4c1c8608-0eb8-402e-b52a-7569f9e66aa2_isr_946074_16.stderr
parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d48f50255138a4ee22f5b4744454701 with result from task 567
parsl.dataflow.dflow DEBUG: Task 567 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 2a202e71808bcb11935b4d4b026d2829
parsl.dataflow.memoization INFO: Task 568 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 567
parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/78ed2c3a-ed82-4f1b-9307-e8b9225ae4aa_characterizeImage_946072_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/78ed2c3a-ed82-4f1b-9307-e8b9225ae4aa_characterizeImage_946072_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ca554f25-3c78-44d8-996f-196beccd552a_isr_946072_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/ca554f25-3c78-44d8-996f-196beccd552a_isr_946072_42.stderr
parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67da7f4e5bbdbee548349da449f3dca0 with result from task 582
parsl.dataflow.dflow DEBUG: Task 582 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 118f6639f62c1feabc8e737b9d1c21a4
parsl.dataflow.memoization INFO: Task 583 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 568
parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7794f967-8a8f-49b8-9ac4-a3c25d3d1fbb_characterizeImage_946182_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7794f967-8a8f-49b8-9ac4-a3c25d3d1fbb_characterizeImage_946182_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c9a103c5-417a-4687-8607-0130c19f87dd_isr_946182_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c9a103c5-417a-4687-8607-0130c19f87dd_isr_946182_33.stderr
parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c79589490fcd02a30408341693c0788f with result from task 585
parsl.dataflow.dflow DEBUG: Task 585 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 7c1040564f2fcfa9b111bb2bc79c5911
parsl.dataflow.memoization INFO: Task 586 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 569
parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f26755f9-85ef-46d4-9b82-21ce4c6cfc3b_characterizeImage_946073_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f26755f9-85ef-46d4-9b82-21ce4c6cfc3b_characterizeImage_946073_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/83ce490a-11bb-4767-87d6-8aeed86070ef_isr_946073_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/83ce490a-11bb-4767-87d6-8aeed86070ef_isr_946073_51.stderr
parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 974deb334c6dbc6c8000df2c1362f17b with result from task 576
parsl.dataflow.dflow DEBUG: Task 576 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 577 has memoization hash dc5943b8dedf5dc58ff84dc9192c5546
parsl.dataflow.memoization INFO: Task 577 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 570
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0d11567f-f42c-4428-81d7-f4c9fdaaf1a4_characterizeImage_946180_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0d11567f-f42c-4428-81d7-f4c9fdaaf1a4_characterizeImage_946180_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4ecd8c40-f41e-4ad3-a1d5-4333fa9b9d83_isr_946180_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/4ecd8c40-f41e-4ad3-a1d5-4333fa9b9d83_isr_946180_27.stderr
parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7310e2889a68806bb467f7421d4fa264 with result from task 579
parsl.dataflow.dflow DEBUG: Task 579 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 79e884194ee90bf37fdfd25c5ac53066
parsl.dataflow.memoization INFO: Task 580 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 571
parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0707ecfc-3a94-45b8-8fbd-453c339c670f_characterizeImage_946072_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0707ecfc-3a94-45b8-8fbd-453c339c670f_characterizeImage_946072_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2633f997-3fd2-481c-b682-60f12d58fb5a_isr_946072_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2633f997-3fd2-481c-b682-60f12d58fb5a_isr_946072_34.stderr
parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc40192f1ac31f64923b9fb733590a3 with result from task 573
parsl.dataflow.dflow DEBUG: Task 573 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 2984f6c53b66b56ed5e709a118e73af3
parsl.dataflow.memoization INFO: Task 574 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 572
parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/485948a4-136e-4ca6-85d7-33e900ce4ac0_characterizeImage_946072_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/485948a4-136e-4ca6-85d7-33e900ce4ac0_characterizeImage_946072_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4cbdaa1e-ec19-408b-9f68-3930ff586fc0_isr_946072_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/4cbdaa1e-ec19-408b-9f68-3930ff586fc0_isr_946072_59.stderr
parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 013ef9ce02bf75ac8d5c16699f952b8e with result from task 588
parsl.dataflow.dflow DEBUG: Task 588 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 4d245511e0ba090bbdaee89a925cb040
parsl.dataflow.memoization INFO: Task 589 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 573
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/ea52618e-a48d-4ef8-aab1-c76f8fcb5ff1_characterizeImage_946072_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/ea52618e-a48d-4ef8-aab1-c76f8fcb5ff1_characterizeImage_946072_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/a21c62c6-8b60-4aa5-8268-50c301fab6ff_isr_946072_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/a21c62c6-8b60-4aa5-8268-50c301fab6ff_isr_946072_39.stderr
parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37f49ae6a45609a41152aad056691dc8 with result from task 1092
parsl.dataflow.dflow DEBUG: Task 1092 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 45695d6886db1a334c42d8b202fbc1b2
parsl.dataflow.memoization INFO: Task 1093 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 574
parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e2dbab3c-a2a5-4efa-825b-16d7b58013fe_characterizeImage_946072_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e2dbab3c-a2a5-4efa-825b-16d7b58013fe_characterizeImage_946072_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c7d1c6b9-ba96-452d-92b3-20046ee31635_isr_946072_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c7d1c6b9-ba96-452d-92b3-20046ee31635_isr_946072_56.stderr
parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c98d74a35e78991043a8c8bb5c97bc5 with result from task 1029
parsl.dataflow.dflow DEBUG: Task 1029 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 4e3f3e00fa394d4a471a109bf161611f
parsl.dataflow.memoization INFO: Task 1030 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 575
parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b1a62023-4b9f-48ae-b786-23c6dd482454_characterizeImage_946182_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b1a62023-4b9f-48ae-b786-23c6dd482454_characterizeImage_946182_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7c19adf3-844b-4715-80a3-72b221f59e9a_isr_946182_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7c19adf3-844b-4715-80a3-72b221f59e9a_isr_946182_55.stderr
parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 48cc9457c1d875ba96661f5d03467354 with result from task 1101
parsl.dataflow.dflow DEBUG: Task 1101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 57cbc3d037e15c5abee31d1ca4b1a1c6
parsl.dataflow.memoization INFO: Task 1102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 576
parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5b6c61f6-7156-4baf-9949-dc2a746f13bc_characterizeImage_946180_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5b6c61f6-7156-4baf-9949-dc2a746f13bc_characterizeImage_946180_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/62647372-819e-4b3c-b727-60171e1b0df0_isr_946180_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/62647372-819e-4b3c-b727-60171e1b0df0_isr_946180_42.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bce608f7636cd3284e527ca61b135955 with result from task 564
parsl.dataflow.dflow DEBUG: Task 564 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 565 has memoization hash ed7ed5a4dfc6bda75c39f5bc24e45579
parsl.dataflow.memoization INFO: Task 565 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 577
parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6a6e1609-8dfc-443c-929c-7eb6b97883d9_characterizeImage_946182_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6a6e1609-8dfc-443c-929c-7eb6b97883d9_characterizeImage_946182_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/41ea4728-fa42-402f-89a7-d4bb200baa90_isr_946182_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/41ea4728-fa42-402f-89a7-d4bb200baa90_isr_946182_15.stderr
parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry acf49bce40428e68331df8c14c8a46e4 with result from task 558
parsl.dataflow.dflow DEBUG: Task 558 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 33fe6a6e462911ebe1cc2140b0b7c14e
parsl.dataflow.memoization INFO: Task 559 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 578
parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/957d6b3e-00b4-4040-835e-bc8fd837c8ca_characterizeImage_946181_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/957d6b3e-00b4-4040-835e-bc8fd837c8ca_characterizeImage_946181_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d695330f-ea22-40bb-9403-559643272de8_isr_946181_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d695330f-ea22-40bb-9403-559643272de8_isr_946181_33.stderr
parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e0172bc4bd0a06400afaee3f86c6de4 with result from task 561
parsl.dataflow.dflow DEBUG: Task 561 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 065f1351e5a5cf42095c0b19fa44a08c
parsl.dataflow.memoization INFO: Task 562 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 579
parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/367d2d53-9658-42e8-bd3b-00ce81ada50b_characterizeImage_946074_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/367d2d53-9658-42e8-bd3b-00ce81ada50b_characterizeImage_946074_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c05ea6f0-1961-4268-98d2-8d3419e5a0a6_isr_946074_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c05ea6f0-1961-4268-98d2-8d3419e5a0a6_isr_946074_30.stderr
parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2537ba2f4ffaf05771ca766d50fb7a67 with result from task 543
parsl.dataflow.dflow DEBUG: Task 543 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 960a1d08e8e3ff0940a38a3db43ac935
parsl.dataflow.memoization INFO: Task 544 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 580
parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fc27e22a-ae6c-4ebd-b334-1d8cd9db0366_characterizeImage_946180_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fc27e22a-ae6c-4ebd-b334-1d8cd9db0366_characterizeImage_946180_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/967c872f-6755-477a-86d9-bfff50f3d7ea_isr_946180_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/967c872f-6755-477a-86d9-bfff50f3d7ea_isr_946180_1.stderr
parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d7976fc5295946bfe0db465aff8d727 with result from task 555
parsl.dataflow.dflow DEBUG: Task 555 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 556 has memoization hash f028390d9c4a5ca157d6c547795ceeae
parsl.dataflow.memoization INFO: Task 556 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 581
parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c7fd7132-c5c4-4e6c-9978-ff3fe2997010_characterizeImage_946182_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c7fd7132-c5c4-4e6c-9978-ff3fe2997010_characterizeImage_946182_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/246d67af-3ac9-4642-b567-0543388569cd_isr_946182_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/246d67af-3ac9-4642-b567-0543388569cd_isr_946182_47.stderr
parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4ef04b88cccd833f3c095275533d30 with result from task 552
parsl.dataflow.dflow DEBUG: Task 552 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 82fc67c6cac65721de634d34a3a07144
parsl.dataflow.memoization INFO: Task 553 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 582
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f514ca-8064-4c2c-958f-68ed4d775c12_characterizeImage_946074_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f514ca-8064-4c2c-958f-68ed4d775c12_characterizeImage_946074_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b7a2be9c-9960-4e3c-8dc9-3e043851a9d8_isr_946074_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b7a2be9c-9960-4e3c-8dc9-3e043851a9d8_isr_946074_22.stderr
parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d441fdeef7f6d6407786c2017bbbaa24 with result from task 549
parsl.dataflow.dflow DEBUG: Task 549 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 2d57dac4ff126f564cba80846409b99e
parsl.dataflow.memoization INFO: Task 550 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 583
parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d93100ac-38b9-492b-8b50-40f0f35fdb08_characterizeImage_946181_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d93100ac-38b9-492b-8b50-40f0f35fdb08_characterizeImage_946181_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2d08a6d2-02da-43de-9cb8-bd50a8cac4e8_isr_946181_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/2d08a6d2-02da-43de-9cb8-bd50a8cac4e8_isr_946181_30.stderr
parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f2cb482a5fcc013a6f40796888a1bb2 with result from task 546
parsl.dataflow.dflow DEBUG: Task 546 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 9996b983a28d981c965ade4a75b98265
parsl.dataflow.memoization INFO: Task 547 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 584
parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b104e42-bd90-4db0-8c74-351b45755143_characterizeImage_946181_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b104e42-bd90-4db0-8c74-351b45755143_characterizeImage_946181_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5cc460c4-2388-413c-8ea5-ecafdb7309ea_isr_946181_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5cc460c4-2388-413c-8ea5-ecafdb7309ea_isr_946181_38.stderr
parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b85e93ccead4e34346306d6faa4cb15 with result from task 309
parsl.dataflow.dflow DEBUG: Task 309 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash b4edfa638a5ad64ab3eaa8f08278dc89
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 585
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5995a2ac-95f6-44f6-a0eb-f73ce9875bd2_characterizeImage_946181_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5995a2ac-95f6-44f6-a0eb-f73ce9875bd2_characterizeImage_946181_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/69a737c6-c9f1-45dc-a201-78c32dbbab6c_isr_946181_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/69a737c6-c9f1-45dc-a201-78c32dbbab6c_isr_946181_3.stderr
parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f3d2ab122ec5ab2ffbbe286e7653ae76 with result from task 291
parsl.dataflow.dflow DEBUG: Task 291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 7a7a89008cfabdd44e9100f6a0973b0a
parsl.dataflow.memoization INFO: Task 292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 586
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a4c94f7-3f67-425a-bf00-b72f0f06a20a_characterizeImage_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a4c94f7-3f67-425a-bf00-b72f0f06a20a_characterizeImage_946073_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b4755169-fedd-4eb5-b736-b13522655465_isr_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/b4755169-fedd-4eb5-b736-b13522655465_isr_946073_61.stderr
parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cfe4ddc5faaae9d9851c135f90f4b645 with result from task 285
parsl.dataflow.dflow DEBUG: Task 285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 286 has memoization hash ee3a947fcb5b60be944d22a125d20d59
parsl.dataflow.memoization INFO: Task 286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 587
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/419feebc-a639-41c6-a913-a7630e35ff13_characterizeImage_946072_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/419feebc-a639-41c6-a913-a7630e35ff13_characterizeImage_946072_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/de6b5108-90a4-4748-8c95-7bc62453f7af_isr_946072_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/de6b5108-90a4-4748-8c95-7bc62453f7af_isr_946072_29.stderr
parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba65c9f5f12b099fd62b3674f28bdbd5 with result from task 288
parsl.dataflow.dflow DEBUG: Task 288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 233e963b5af5dc7fff47eee3905ea9f4
parsl.dataflow.memoization INFO: Task 289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 588
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/04024df1-8589-4c48-8dcf-6f1c1e951a59_characterizeImage_946180_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/04024df1-8589-4c48-8dcf-6f1c1e951a59_characterizeImage_946180_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a7d03e77-670a-4887-85a5-31d2f9da95cc_isr_946180_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a7d03e77-670a-4887-85a5-31d2f9da95cc_isr_946180_44.stderr
parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 567e92f5b30de4575ba9798d130424d5 with result from task 300
parsl.dataflow.dflow DEBUG: Task 300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 38c6a169ae2811920ef319f6aeeb09db
parsl.dataflow.memoization INFO: Task 301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 589
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/888d288f-dc6f-4cf3-95d3-1d7fe9202cbe_characterizeImage_946182_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/888d288f-dc6f-4cf3-95d3-1d7fe9202cbe_characterizeImage_946182_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2e0c1bb3-f968-4ab4-8f11-4bfd131a8c9b_isr_946182_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2e0c1bb3-f968-4ab4-8f11-4bfd131a8c9b_isr_946182_59.stderr
parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2559ee7238175cda75041b819d592d2b with result from task 315
parsl.dataflow.dflow DEBUG: Task 315 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 8c5d29faea73e66d40ed9c4898d7a029
parsl.dataflow.memoization INFO: Task 316 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 590
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/997fb956-ed0c-456e-becc-3aa0c88201fd_characterizeImage_946180_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/997fb956-ed0c-456e-becc-3aa0c88201fd_characterizeImage_946180_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c825bfa7-d14a-4e30-868e-34bed79118bd_isr_946180_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c825bfa7-d14a-4e30-868e-34bed79118bd_isr_946180_46.stderr
parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79b31e1c0036571bc7c769711b13fecc with result from task 306
parsl.dataflow.dflow DEBUG: Task 306 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash f4c15d70b8bf7ca4d093fb920f44bb8e
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 591
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7aaead84-de07-4093-ba17-6eb162a1b867_characterizeImage_946074_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7aaead84-de07-4093-ba17-6eb162a1b867_characterizeImage_946074_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ae576817-f9f0-4f64-a53b-0302f9afb70e_isr_946074_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ae576817-f9f0-4f64-a53b-0302f9afb70e_isr_946074_53.stderr
parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a921c85cd12ea5e006e851f29c48c35 with result from task 312
parsl.dataflow.dflow DEBUG: Task 312 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 3123b0a87476f8b42bf390bd6e8760b9
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 592
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/61babb40-3c60-48d4-888c-011075ee2598_characterizeImage_946074_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/61babb40-3c60-48d4-888c-011075ee2598_characterizeImage_946074_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ee2b8646-f5cc-450c-be1c-7a56887999d0_isr_946074_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ee2b8646-f5cc-450c-be1c-7a56887999d0_isr_946074_11.stderr
parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a86381f43fd0be69e8196fd3e9d7fe6 with result from task 297
parsl.dataflow.dflow DEBUG: Task 297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 3152b58db1da8f4e4f999b6851da97d3
parsl.dataflow.memoization INFO: Task 298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 593
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/64b03aa7-fb54-45cd-89c0-68f04970a9bf_characterizeImage_946180_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/64b03aa7-fb54-45cd-89c0-68f04970a9bf_characterizeImage_946180_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/bb97410f-f6f1-4a5c-9b0c-7691a16d2f35_isr_946180_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/bb97410f-f6f1-4a5c-9b0c-7691a16d2f35_isr_946180_52.stderr
parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cdc64e0bd5fdb0003edc8cea0a1aaaa9 with result from task 294
parsl.dataflow.dflow DEBUG: Task 294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 53e4f30f93884335446e4550d7380b26
parsl.dataflow.memoization INFO: Task 295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 594
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/05e17be8-4d28-452a-aca2-16a1103991e4_characterizeImage_946072_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/05e17be8-4d28-452a-aca2-16a1103991e4_characterizeImage_946072_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/39337a3c-c2e1-4eba-9980-a643e34a245e_isr_946072_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/39337a3c-c2e1-4eba-9980-a643e34a245e_isr_946072_41.stderr
parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab9b00e6b8d690a701792ee57277e90 with result from task 303
parsl.dataflow.dflow DEBUG: Task 303 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 64b5445ec17c4536f09463b5ee85edfd
parsl.dataflow.memoization INFO: Task 304 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 595
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/688376e5-ca43-4984-bf39-f37c19ad65bd_characterizeImage_946182_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/688376e5-ca43-4984-bf39-f37c19ad65bd_characterizeImage_946182_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/df29ff19-ac0d-471d-a6e3-f57189501125_isr_946182_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/df29ff19-ac0d-471d-a6e3-f57189501125_isr_946182_4.stderr
parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 390263a1ec7bc3ef44926d7642ae6270 with result from task 531
parsl.dataflow.dflow DEBUG: Task 531 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 32bdb0a111a577e3ae3140d6c1ac8afb
parsl.dataflow.memoization INFO: Task 532 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 596
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2b7cc1f2-c802-4055-ab68-8cc8480fa471_characterizeImage_946182_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2b7cc1f2-c802-4055-ab68-8cc8480fa471_characterizeImage_946182_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b47f35a-5a8d-42de-91a1-dca4ad2c6cdb_isr_946182_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b47f35a-5a8d-42de-91a1-dca4ad2c6cdb_isr_946182_54.stderr
parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90b33b2ce9a4b6765579ba19d3bffe2d with result from task 525
parsl.dataflow.dflow DEBUG: Task 525 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 526 has memoization hash b0e9ed190a283bc15aaa5af3b797cddb
parsl.dataflow.memoization INFO: Task 526 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 597
parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/35e59a41-4f72-458e-88bf-65b950899c52_characterizeImage_946074_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/35e59a41-4f72-458e-88bf-65b950899c52_characterizeImage_946074_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/55c10de7-c629-47d2-a312-c0c488048523_isr_946074_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/55c10de7-c629-47d2-a312-c0c488048523_isr_946074_60.stderr
parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd411af703521dc4f5e54a8b7e0fb9b9 with result from task 519
parsl.dataflow.dflow DEBUG: Task 519 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 4a712779b61cac41f48f6f61c46ebba8
parsl.dataflow.memoization INFO: Task 520 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 598
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/42f58a95-05c1-4ce0-9db3-55ba00d47fe7_characterizeImage_946073_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/42f58a95-05c1-4ce0-9db3-55ba00d47fe7_characterizeImage_946073_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/634c0665-d966-45f3-8bf2-bb47e605940a_isr_946073_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/634c0665-d966-45f3-8bf2-bb47e605940a_isr_946073_24.stderr
parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6faa7c2974d146ddec5f59a36a2c10f with result from task 534
parsl.dataflow.dflow DEBUG: Task 534 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 29bd8d15a422f5edaea1f9ca2a6c744e
parsl.dataflow.memoization INFO: Task 535 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 599
parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/a06f6976-ec38-4626-ac60-54bc5b825153_characterizeImage_946074_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/a06f6976-ec38-4626-ac60-54bc5b825153_characterizeImage_946074_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/97c84d08-d489-40ca-9f12-9bcd5e6f6564_isr_946074_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/97c84d08-d489-40ca-9f12-9bcd5e6f6564_isr_946074_41.stderr
parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a254a8cb7eed5dbce489be01f7b769ec with result from task 540
parsl.dataflow.dflow DEBUG: Task 540 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 541 has memoization hash ea5fb4a9b2ad647a362f8c4d977c6d5d
parsl.dataflow.memoization INFO: Task 541 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 600
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/500cd511-2935-4b25-a5c4-fd88cada8e6d_characterizeImage_946072_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/500cd511-2935-4b25-a5c4-fd88cada8e6d_characterizeImage_946072_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6858fbcf-1771-4358-bfe4-42d9ab39b902_isr_946072_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/6858fbcf-1771-4358-bfe4-42d9ab39b902_isr_946072_25.stderr
parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b82c4aa654529ffc8f31a1287f834874 with result from task 522
parsl.dataflow.dflow DEBUG: Task 522 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 30f9e8b362f71480027c428386361d06
parsl.dataflow.memoization INFO: Task 523 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 601
parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/aad2c7b1-b8ed-42df-9daa-6ad4ac833ce5_characterizeImage_946181_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/aad2c7b1-b8ed-42df-9daa-6ad4ac833ce5_characterizeImage_946181_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3f30c75b-6ef4-4f41-84a3-7eb9a24989aa_isr_946181_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/3f30c75b-6ef4-4f41-84a3-7eb9a24989aa_isr_946181_43.stderr
parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50a36a0ed1cbd8e18a2338acc93bfbc0 with result from task 528
parsl.dataflow.dflow DEBUG: Task 528 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 0b5a2dda7dc6f0296616e9874fb38837
parsl.dataflow.memoization INFO: Task 529 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 602
parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ca9bf1a9-12ed-4b27-90ba-2ec9566c9f8a_characterizeImage_946073_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ca9bf1a9-12ed-4b27-90ba-2ec9566c9f8a_characterizeImage_946073_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dddc5323-cca0-42bc-8383-4b409a817dbd_isr_946073_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/dddc5323-cca0-42bc-8383-4b409a817dbd_isr_946073_34.stderr
parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a8df1c43668d292740e05eb2c43a00dc with result from task 537
parsl.dataflow.dflow DEBUG: Task 537 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 538 has memoization hash ecd661160f83f40f94e0fcf9aa1271fb
parsl.dataflow.memoization INFO: Task 538 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 603
parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c576b14d-7b11-47f3-9ce9-23c510c1d6d5_characterizeImage_946180_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c576b14d-7b11-47f3-9ce9-23c510c1d6d5_characterizeImage_946180_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2f35693f-9d01-4cd5-893f-888200387d27_isr_946180_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2f35693f-9d01-4cd5-893f-888200387d27_isr_946180_28.stderr
parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd3bc14476af18f74f644fa50e6205d8 with result from task 825
parsl.dataflow.dflow DEBUG: Task 825 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 68ba42d7c7d1873deb7ead96f130b702
parsl.dataflow.memoization INFO: Task 826 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 604
parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5c2fcdb3-d753-4e0b-8b97-2afa6b0e86a3_characterizeImage_946181_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5c2fcdb3-d753-4e0b-8b97-2afa6b0e86a3_characterizeImage_946181_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bc01ed06-0486-46fc-9656-8f539d3c460a_isr_946181_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bc01ed06-0486-46fc-9656-8f539d3c460a_isr_946181_19.stderr
parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8089a528dbcd9521a90b3179d113635 with result from task 837
parsl.dataflow.dflow DEBUG: Task 837 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 838 has memoization hash ccd9e429f0b45929ab262f230c4c46e8
parsl.dataflow.memoization INFO: Task 838 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 605
parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/560d8e4c-61f4-4456-ad7c-c398cf0457aa_characterizeImage_946182_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/560d8e4c-61f4-4456-ad7c-c398cf0457aa_characterizeImage_946182_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2cae4652-eeb8-4124-adc5-83f6d46d0501_isr_946182_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2cae4652-eeb8-4124-adc5-83f6d46d0501_isr_946182_40.stderr
parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed43a283fe72594061998ddfee93d363 with result from task 828
parsl.dataflow.dflow DEBUG: Task 828 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 4bc7fee1123aa1606fc4bab7ad4de8fe
parsl.dataflow.memoization INFO: Task 829 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 606
parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6910a32e-99e1-4300-bcfd-e31a4e5600c6_characterizeImage_946180_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6910a32e-99e1-4300-bcfd-e31a4e5600c6_characterizeImage_946180_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e37df7a0-b464-4031-8c26-a2712f9b4c5a_isr_946180_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/e37df7a0-b464-4031-8c26-a2712f9b4c5a_isr_946180_50.stderr
parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d14da21d50d7ae42485815f6fa7d6ee5 with result from task 843
parsl.dataflow.dflow DEBUG: Task 843 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 1e3dd7afa62397f29a6bd4dceabb6a6a
parsl.dataflow.memoization INFO: Task 844 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 607
parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a64d37d6-c6cc-440f-afea-3c99994ad344_characterizeImage_946181_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a64d37d6-c6cc-440f-afea-3c99994ad344_characterizeImage_946181_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/65a573ed-83de-4c08-91db-8be7203525df_isr_946181_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/65a573ed-83de-4c08-91db-8be7203525df_isr_946181_27.stderr
parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 342c6cd54b9a05cebb8674a9ee8156ad with result from task 840
parsl.dataflow.dflow DEBUG: Task 840 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 3369ec1c9c65fb11aa7253799690dc12
parsl.dataflow.memoization INFO: Task 841 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 608
parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3517e8d9-fc98-43c4-8cef-4e7eec5dafc5_characterizeImage_946074_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3517e8d9-fc98-43c4-8cef-4e7eec5dafc5_characterizeImage_946074_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1ae41411-f3ef-43cf-b42a-f75e658bc7b3_isr_946074_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/1ae41411-f3ef-43cf-b42a-f75e658bc7b3_isr_946074_47.stderr
parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24be507e6ed1072ff8ce777720ffc72f with result from task 831
parsl.dataflow.dflow DEBUG: Task 831 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 0ac0b2d2aba104468b4a4b3bc3fef793
parsl.dataflow.memoization INFO: Task 832 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 609
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8bc4e169-0e5a-489a-9f31-5e825ff2d3b6_characterizeImage_946182_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8bc4e169-0e5a-489a-9f31-5e825ff2d3b6_characterizeImage_946182_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ccb02535-7def-45af-9a4e-2e794439e079_isr_946182_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/ccb02535-7def-45af-9a4e-2e794439e079_isr_946182_1.stderr
parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc1f23a721bce796250c7db6939c89a3 with result from task 822
parsl.dataflow.dflow DEBUG: Task 822 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 39f7ee7ca594eb1d8120aed1c8ebf6db
parsl.dataflow.memoization INFO: Task 823 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 610
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/cd9fd5ba-f5ae-4559-992e-795aca3b645e_characterizeImage_946180_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/cd9fd5ba-f5ae-4559-992e-795aca3b645e_characterizeImage_946180_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/88f6d1a2-46e0-4868-abe6-c8f965364ab2_isr_946180_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/88f6d1a2-46e0-4868-abe6-c8f965364ab2_isr_946180_47.stderr
parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9b0532d878846d3d17d6e0d833ac3cd9 with result from task 834
parsl.dataflow.dflow DEBUG: Task 834 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 866278259ae65d5b22de3810ff5c6857
parsl.dataflow.memoization INFO: Task 835 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 611
parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/f1f0f7b1-5ec6-471a-8ec5-066628a3d5de_characterizeImage_946074_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/f1f0f7b1-5ec6-471a-8ec5-066628a3d5de_characterizeImage_946074_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/2bab4eee-6f6f-4c0d-9686-08d2bd86f5b5_isr_946074_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/2bab4eee-6f6f-4c0d-9686-08d2bd86f5b5_isr_946074_32.stderr
parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 871ea64e2f8e468f328bdcdb779e7e04 with result from task 501
parsl.dataflow.dflow DEBUG: Task 501 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 657200f68fd0117c8ab3bbef4a3097c7
parsl.dataflow.memoization INFO: Task 502 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 612
parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e9c9115c-6283-4bbe-9edf-0129ba38c856_characterizeImage_946181_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e9c9115c-6283-4bbe-9edf-0129ba38c856_characterizeImage_946181_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7b2159e5-9437-4f1e-8faf-25e960a35a9c_isr_946181_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7b2159e5-9437-4f1e-8faf-25e960a35a9c_isr_946181_56.stderr
parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 481347e9143b182fec7d5c5c2e9e9754 with result from task 507
parsl.dataflow.dflow DEBUG: Task 507 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 1dc3edd88df69bf04b980e6d29c3191e
parsl.dataflow.memoization INFO: Task 508 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 613
parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ed9d1294-2e41-4959-b1f6-9705206f78a9_characterizeImage_946182_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ed9d1294-2e41-4959-b1f6-9705206f78a9_characterizeImage_946182_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a27f0bd3-a495-4957-9dae-99815f6f90cc_isr_946182_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a27f0bd3-a495-4957-9dae-99815f6f90cc_isr_946182_10.stderr
parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0b79bea324c83022fc9a804fd3cd4e7 with result from task 513
parsl.dataflow.dflow DEBUG: Task 513 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 18f75ebc8a58aaf9c8c70b80f8f9e56e
parsl.dataflow.memoization INFO: Task 514 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 614
parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9bbbe97c-eff9-4393-8c07-b48be2d482de_characterizeImage_946072_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9bbbe97c-eff9-4393-8c07-b48be2d482de_characterizeImage_946072_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/53289755-253a-4d99-8a76-cf11866676d7_isr_946072_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/53289755-253a-4d99-8a76-cf11866676d7_isr_946072_46.stderr
parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 406d05206a42a0160d341d0ca55e864e with result from task 510
parsl.dataflow.dflow DEBUG: Task 510 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 0f180a03d1763b08c822a70ba23024f4
parsl.dataflow.memoization INFO: Task 511 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 615
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/fdbd1016-9fce-4141-a81a-81360ddb9a41_characterizeImage_946073_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/fdbd1016-9fce-4141-a81a-81360ddb9a41_characterizeImage_946073_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/c11419ff-07c4-4f34-a216-34ccf5dcb1e1_isr_946073_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/c11419ff-07c4-4f34-a216-34ccf5dcb1e1_isr_946073_17.stderr
parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfe9f0e9b44784c9b38dc1247497026d with result from task 504
parsl.dataflow.dflow DEBUG: Task 504 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 505 has memoization hash fcc126503d05f990a788f409c3032855
parsl.dataflow.memoization INFO: Task 505 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 616
parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7965a59-990d-46ff-a84d-e5f71644da48_characterizeImage_946181_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7965a59-990d-46ff-a84d-e5f71644da48_characterizeImage_946181_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bd8d946a-d286-4e80-a768-1de5a9834f7c_isr_946181_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/bd8d946a-d286-4e80-a768-1de5a9834f7c_isr_946181_49.stderr
parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 538e56242c45457829bb80715cff9104 with result from task 495
parsl.dataflow.dflow DEBUG: Task 495 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 496 has memoization hash b88cbb80e76d21b66ac1c414703b0c61
parsl.dataflow.memoization INFO: Task 496 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 617
parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8d735523-99bf-4204-b6e7-d19a1c18ecf5_characterizeImage_946180_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8d735523-99bf-4204-b6e7-d19a1c18ecf5_characterizeImage_946180_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/82cdd83d-7712-4f80-a42a-2d58fa898785_isr_946180_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/82cdd83d-7712-4f80-a42a-2d58fa898785_isr_946180_18.stderr
parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1ff37add86bace2d26365df84ec7a4 with result from task 516
parsl.dataflow.dflow DEBUG: Task 516 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 517 has memoization hash a0f72a0946deb87c68e52daf6855f2a9
parsl.dataflow.memoization INFO: Task 517 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 618
parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c5cdd94a-7439-4651-a91d-121d2c934afa_characterizeImage_946074_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c5cdd94a-7439-4651-a91d-121d2c934afa_characterizeImage_946074_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/492e42be-127c-4459-98a8-6c1b3b94b0db_isr_946074_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/492e42be-127c-4459-98a8-6c1b3b94b0db_isr_946074_3.stderr
parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a1502ef6109419472dd8e8e3f83ffb45 with result from task 498
parsl.dataflow.dflow DEBUG: Task 498 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 36b2470f087edb0e12e4d94893b6d049
parsl.dataflow.memoization INFO: Task 499 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 619
parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2abf49d3-dc0a-4529-83a6-b7379aa605a5_characterizeImage_946180_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2abf49d3-dc0a-4529-83a6-b7379aa605a5_characterizeImage_946180_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/1b6903d9-cad3-4b5b-8247-75bdc25180cb_isr_946180_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/1b6903d9-cad3-4b5b-8247-75bdc25180cb_isr_946180_22.stderr
parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 608904828061eb78457164dcd906fa4b with result from task 609
parsl.dataflow.dflow DEBUG: Task 609 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 610 has memoization hash c72e7ce086c29ce87447fb99659d9d88
parsl.dataflow.memoization INFO: Task 610 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 620
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/dcb11e24-61b8-49cd-9be2-e66d25b2bdd8_isr_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/dcb11e24-61b8-49cd-9be2-e66d25b2bdd8_isr_946072_2.stderr
parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1438a622c22bcdfcfbbd643a4144f19b with result from task 600
parsl.dataflow.dflow DEBUG: Task 600 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 3469fadc4511aaa56acdd47a925b3930
parsl.dataflow.memoization INFO: Task 601 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 621
parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a8e3b530-91e6-4533-bf5b-2aea2e767769_characterizeImage_946180_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a8e3b530-91e6-4533-bf5b-2aea2e767769_characterizeImage_946180_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/95774431-d4ad-45b3-83f1-3f752f27fea4_isr_946180_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/95774431-d4ad-45b3-83f1-3f752f27fea4_isr_946180_10.stderr
parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da8e8210aa6a4585112919da5b39795f with result from task 678
parsl.dataflow.dflow DEBUG: Task 678 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 20a2788639c2750e79817c710fc70463
parsl.dataflow.memoization INFO: Task 679 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 622
parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9ff2e47f-e0b8-4ae9-b73f-88d7bd58a72c_characterizeImage_946072_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9ff2e47f-e0b8-4ae9-b73f-88d7bd58a72c_characterizeImage_946072_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/066c93c4-919b-43ac-a81d-f2d3a3a3bf8f_isr_946072_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/066c93c4-919b-43ac-a81d-f2d3a3a3bf8f_isr_946072_4.stderr
parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47943a83766f6bcf8f8a3d79d7b5bf5b with result from task 684
parsl.dataflow.dflow DEBUG: Task 684 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 685 has memoization hash f418e2bd8881aa66c473cad13bf04eec
parsl.dataflow.memoization INFO: Task 685 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 623
parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/be98b632-499f-4cfa-99c4-1ccb8472bfae_characterizeImage_946072_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/be98b632-499f-4cfa-99c4-1ccb8472bfae_characterizeImage_946072_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/07dd00d6-f267-45ce-9be9-960729ce0e87_isr_946072_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/07dd00d6-f267-45ce-9be9-960729ce0e87_isr_946072_27.stderr
parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80eecd97cf449ddb76fb7554606caf25 with result from task 591
parsl.dataflow.dflow DEBUG: Task 591 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 43000d8fc74a289be31a77d09382346b
parsl.dataflow.memoization INFO: Task 592 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 624
parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b7bb503d-b30f-4994-a3b4-231ff7bf538b_characterizeImage_946181_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b7bb503d-b30f-4994-a3b4-231ff7bf538b_characterizeImage_946181_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/605d5350-96d6-415a-a3c5-c5120346e167_isr_946181_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/605d5350-96d6-415a-a3c5-c5120346e167_isr_946181_10.stderr
parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce69257568bfe5f5fb9eefdfd4bf1d66 with result from task 666
parsl.dataflow.dflow DEBUG: Task 666 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 53c2a1badcb7e114c042d22825af55fb
parsl.dataflow.memoization INFO: Task 667 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 625
parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/577ed6c1-358b-4b7a-b5c1-c92e61ae8798_characterizeImage_946072_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/577ed6c1-358b-4b7a-b5c1-c92e61ae8798_characterizeImage_946072_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e30c92df-d569-4cb7-9198-320c124edeef_isr_946072_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/e30c92df-d569-4cb7-9198-320c124edeef_isr_946072_17.stderr
parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2d086b3b9ac785839d4077400d4110c with result from task 669
parsl.dataflow.dflow DEBUG: Task 669 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 9951e0fd641f7e2d7caa6b67ce1a743e
parsl.dataflow.memoization INFO: Task 670 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 626
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fc637ec4-c655-4ed1-9f92-7e9fc02ddc9e_characterizeImage_946182_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fc637ec4-c655-4ed1-9f92-7e9fc02ddc9e_characterizeImage_946182_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2492a8e2-3b55-47d1-8d7c-45ef9b871653_isr_946182_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2492a8e2-3b55-47d1-8d7c-45ef9b871653_isr_946182_58.stderr
parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eec59dde2a69ef53496cf201d9056c62 with result from task 594
parsl.dataflow.dflow DEBUG: Task 594 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 3e5a0d05b3b410ff3f5b39f1ac7be357
parsl.dataflow.memoization INFO: Task 595 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 627
parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0da657d1-1cdb-49c7-a997-5cbe1548b194_characterizeImage_946182_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0da657d1-1cdb-49c7-a997-5cbe1548b194_characterizeImage_946182_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c97a06d4-a915-4c57-8c6d-690b7457596b_isr_946182_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c97a06d4-a915-4c57-8c6d-690b7457596b_isr_946182_18.stderr
parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 840e9c7015e3dc9354b41a3ababe829d with result from task 612
parsl.dataflow.dflow DEBUG: Task 612 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 613 has memoization hash a4be02e23634d89653b1f2c42a7e5b18
parsl.dataflow.memoization INFO: Task 613 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 628
parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/788caf51-6ecd-4b02-a70d-744dd2624761_characterizeImage_946182_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/788caf51-6ecd-4b02-a70d-744dd2624761_characterizeImage_946182_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58b3760b-b1b0-4aa2-a14e-9e6465bca753_isr_946182_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/58b3760b-b1b0-4aa2-a14e-9e6465bca753_isr_946182_34.stderr
parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d15dab5043a70cb683f2aa443d9b35af with result from task 606
parsl.dataflow.dflow DEBUG: Task 606 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 76b559113486a13aa239911c3ee57c2c
parsl.dataflow.memoization INFO: Task 607 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 629
parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/64eb9a45-bc10-43d1-95b1-2a371f4a25a0_characterizeImage_946182_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/64eb9a45-bc10-43d1-95b1-2a371f4a25a0_characterizeImage_946182_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c51b1175-bf04-4e66-b0da-761164b00911_isr_946182_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/c51b1175-bf04-4e66-b0da-761164b00911_isr_946182_45.stderr
parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9001c18ad2a10e39bd49b2cdab7ecaa with result from task 675
parsl.dataflow.dflow DEBUG: Task 675 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 2497f6a3b5970c10225d0a2ec1b38586
parsl.dataflow.memoization INFO: Task 676 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 630
parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e8d73969-9155-4930-a0b1-b8b2745e2933_characterizeImage_946074_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e8d73969-9155-4930-a0b1-b8b2745e2933_characterizeImage_946074_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/0d2b5a11-d143-4aa9-80ea-2a3a439ee7a1_isr_946074_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/0d2b5a11-d143-4aa9-80ea-2a3a439ee7a1_isr_946074_1.stderr
parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c83ea403ad92304a6f649c1ee5ea62d5 with result from task 681
parsl.dataflow.dflow DEBUG: Task 681 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 7173a9512812d7ce587a9d50fba6b700
parsl.dataflow.memoization INFO: Task 682 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 631
parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8c136890-3413-4bfe-a8fd-c6ba6fb6c3f9_characterizeImage_946072_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8c136890-3413-4bfe-a8fd-c6ba6fb6c3f9_characterizeImage_946072_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fd2abaed-e0b2-43f5-b21c-ec68af16d511_isr_946072_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/fd2abaed-e0b2-43f5-b21c-ec68af16d511_isr_946072_22.stderr
parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d120a66e19db6c5576a0e09b8bbd238 with result from task 603
parsl.dataflow.dflow DEBUG: Task 603 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 604 has memoization hash cee1d03a99a3540097472dc19b5e47a9
parsl.dataflow.memoization INFO: Task 604 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 632
parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c6e3b646-5a78-4d2e-ae07-28e044365f8c_characterizeImage_946072_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c6e3b646-5a78-4d2e-ae07-28e044365f8c_characterizeImage_946072_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bd68158a-66e0-4235-a1a5-31fd9c0b9296_isr_946072_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bd68158a-66e0-4235-a1a5-31fd9c0b9296_isr_946072_23.stderr
parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d3f28db919a23283342df260782db4c with result from task 597
parsl.dataflow.dflow DEBUG: Task 597 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 598 has memoization hash fa5e961ad7b0a2c3bce7e3f0a4099c62
parsl.dataflow.memoization INFO: Task 598 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 633
parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/84772ee4-b317-40ab-a889-6ea09b32e029_characterizeImage_946074_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/84772ee4-b317-40ab-a889-6ea09b32e029_characterizeImage_946074_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9de20dbe-4ed3-4fb1-9597-64b4791866af_isr_946074_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9de20dbe-4ed3-4fb1-9597-64b4791866af_isr_946074_40.stderr
parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 301442581eb4289cee93a54747d9bd63 with result from task 663
parsl.dataflow.dflow DEBUG: Task 663 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 664 has memoization hash ca4bab1d8dd76b3b599c709cde8dde0f
parsl.dataflow.memoization INFO: Task 664 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 634
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/68b27ae8-b2ee-483a-a27a-7f8fd208a0bf_characterizeImage_946180_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/68b27ae8-b2ee-483a-a27a-7f8fd208a0bf_characterizeImage_946180_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/03c352fd-0a18-43fa-bd0d-d4d5d9d50d0a_isr_946180_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/03c352fd-0a18-43fa-bd0d-d4d5d9d50d0a_isr_946180_11.stderr
parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ed5c0a1b193a7cdca97a5dea2f141a2 with result from task 672
parsl.dataflow.dflow DEBUG: Task 672 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 5de0e558736fecf308316c02222ee11d
parsl.dataflow.memoization INFO: Task 673 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 635
parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a38f5c90-94f0-494b-b2cb-4b63cf74fb89_characterizeImage_946072_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a38f5c90-94f0-494b-b2cb-4b63cf74fb89_characterizeImage_946072_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c483220b-2763-4ada-ae7f-ee9aeaab9de3_isr_946072_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c483220b-2763-4ada-ae7f-ee9aeaab9de3_isr_946072_43.stderr
parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09d1b2e54106462971bc7a24ad81a7ee with result from task 390
parsl.dataflow.dflow DEBUG: Task 390 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 391 has memoization hash ecd412fee0560e965887de5fb2720278
parsl.dataflow.memoization INFO: Task 391 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 636
parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/b03e2b6a-49df-4351-aecb-d9215e1ebc1a_characterizeImage_946072_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/b03e2b6a-49df-4351-aecb-d9215e1ebc1a_characterizeImage_946072_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d3f4f9bd-64ff-4bd9-b41d-f5b322aafbf9_isr_946072_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/d3f4f9bd-64ff-4bd9-b41d-f5b322aafbf9_isr_946072_21.stderr
parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 193158e2e270ddcabf05c85642d21c25 with result from task 417
parsl.dataflow.dflow DEBUG: Task 417 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 418 has memoization hash d9db19a50defcd33efd43d7b5885e584
parsl.dataflow.memoization INFO: Task 418 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 637
parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8bb9db62-4eca-459e-b094-97eb8e1c0ed7_characterizeImage_946072_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8bb9db62-4eca-459e-b094-97eb8e1c0ed7_characterizeImage_946072_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/7c4a1f2c-4cb9-4ffd-8a21-a0c6a34b0b0f_isr_946072_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/7c4a1f2c-4cb9-4ffd-8a21-a0c6a34b0b0f_isr_946072_54.stderr
parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b54aca5532ae3090eedc6a317153c488 with result from task 420
parsl.dataflow.dflow DEBUG: Task 420 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 421 has memoization hash d78da5c2a5aebe20405ebd279bf9a5ea
parsl.dataflow.memoization INFO: Task 421 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 638
parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5215bcec-ad78-4df0-9714-a1580e24986d_characterizeImage_946072_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5215bcec-ad78-4df0-9714-a1580e24986d_characterizeImage_946072_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9b8e3c22-13b1-46e1-83bf-85f72ed093eb_isr_946072_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9b8e3c22-13b1-46e1-83bf-85f72ed093eb_isr_946072_35.stderr
2024-03-13 21:36:21 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:36:21 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:21 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:22 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:22 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:22 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:22 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9259d1238282fda1b1cce336c051e829 with result from task 331
parsl.dataflow.dflow DEBUG: Task 331 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 89e36bd77245a240e703bd7b706e5b04
parsl.dataflow.memoization INFO: Task 332 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 639
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b31fcdd0-de93-4f5f-b06d-457df1214255_calibrate_946072_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b31fcdd0-de93-4f5f-b06d-457df1214255_calibrate_946072_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1c1174bd-eafd-4c07-98f0-d11b82ae8d83_characterizeImage_946072_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1c1174bd-eafd-4c07-98f0-d11b82ae8d83_characterizeImage_946072_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:23 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:23 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9779ebb141d7a3958651c76b1e96a89 with result from task 382
parsl.dataflow.dflow DEBUG: Task 382 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 2c44778d4174e164eb24adf419efd6bf
parsl.dataflow.memoization INFO: Task 383 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 640
parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a02c7b00-1b74-4796-918c-b0a1183a7434_calibrate_946073_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a02c7b00-1b74-4796-918c-b0a1183a7434_calibrate_946073_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e782443c-88fc-4a3f-8301-77e1d2c1ff51_characterizeImage_946073_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e782443c-88fc-4a3f-8301-77e1d2c1ff51_characterizeImage_946073_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:23 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6063d6f9dc2926f418d50f05b6828cf6 with result from task 471
parsl.dataflow.dflow DEBUG: Task 471 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 2cf57bdd73d35dcaa06749adcdd3c6de
parsl.dataflow.memoization INFO: Task 472 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 641
parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1725caa-c8bc-447f-ac63-cc399b0d459c_characterizeImage_946073_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1725caa-c8bc-447f-ac63-cc399b0d459c_characterizeImage_946073_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/5f644685-f5ee-410d-85e5-e02ef9534af0_isr_946073_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/5f644685-f5ee-410d-85e5-e02ef9534af0_isr_946073_62.stderr
parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80a103817df8643e49f4b3b9d830cc4b with result from task 453
parsl.dataflow.dflow DEBUG: Task 453 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 454 has memoization hash c99cccc3adc42bb93e5dbd40e5d304c6
parsl.dataflow.memoization INFO: Task 454 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 642
parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9398606f-9a50-4f31-818a-1680ef51858b_characterizeImage_946181_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9398606f-9a50-4f31-818a-1680ef51858b_characterizeImage_946181_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9b494846-9744-4535-9ecc-09a85a5a7986_isr_946181_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9b494846-9744-4535-9ecc-09a85a5a7986_isr_946181_52.stderr
parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89489ab826d62a10080831f6f02d6a7e with result from task 450
parsl.dataflow.dflow DEBUG: Task 450 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 11ee58d47f5c92cb7c81a6c522e27b2d
parsl.dataflow.memoization INFO: Task 451 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 643
parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0f97984b-928e-4a1a-b60a-867b572eb572_characterizeImage_946181_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0f97984b-928e-4a1a-b60a-867b572eb572_characterizeImage_946181_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e4feb5a6-5366-472b-85d9-6032dc78e1b0_isr_946181_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e4feb5a6-5366-472b-85d9-6032dc78e1b0_isr_946181_53.stderr
parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 17a5fe1c8cbe9e13d3e28a1e82cb11d5 with result from task 468
parsl.dataflow.dflow DEBUG: Task 468 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 155928f0fc76de5eb6714ff12279e90c
parsl.dataflow.memoization INFO: Task 469 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 644
parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8998ca5c-b4c0-41f3-9649-8ffc637d5966_characterizeImage_946182_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8998ca5c-b4c0-41f3-9649-8ffc637d5966_characterizeImage_946182_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0ab26328-4b79-4914-9a2d-8c7e087612cb_isr_946182_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0ab26328-4b79-4914-9a2d-8c7e087612cb_isr_946182_35.stderr
parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe582d2dc4b43e40d1bcfecefbda0a8e with result from task 187
parsl.dataflow.dflow DEBUG: Task 187 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 67acecee43ae2f502be8f18a65d58b36
parsl.dataflow.memoization INFO: Task 188 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 645
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/55a2c953-0556-4c32-80b8-25e250833647_calibrate_946073_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/55a2c953-0556-4c32-80b8-25e250833647_calibrate_946073_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b4c732b-5048-4abf-bc39-1ae4c0165aff_characterizeImage_946073_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b4c732b-5048-4abf-bc39-1ae4c0165aff_characterizeImage_946073_21.stderr
parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7723327fecc40d9f80cee257dc9caeed with result from task 486
parsl.dataflow.dflow DEBUG: Task 486 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 2ce630ed93644f88074b70b4e73fcc66
parsl.dataflow.memoization INFO: Task 487 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 646
parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/038350f3-683b-40ba-9e53-a83a8e26715a_characterizeImage_946074_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/038350f3-683b-40ba-9e53-a83a8e26715a_characterizeImage_946074_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c54a0c1d-878d-4ead-828c-ad1f5f3335fd_isr_946074_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c54a0c1d-878d-4ead-828c-ad1f5f3335fd_isr_946074_39.stderr
parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59fe389db4f03b67156d0f0653452a7b with result from task 480
parsl.dataflow.dflow DEBUG: Task 480 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 62d1647a09c23b7a747b5e14952fbfee
parsl.dataflow.memoization INFO: Task 481 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 647
parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/dbfd4482-be8e-413c-a8c9-4490a42a06d7_characterizeImage_946180_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/dbfd4482-be8e-413c-a8c9-4490a42a06d7_characterizeImage_946180_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c01b65b3-7c1b-4569-906e-30e9443a201a_isr_946180_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/c01b65b3-7c1b-4569-906e-30e9443a201a_isr_946180_20.stderr
parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0a2a076f3281a65738854fa847ad9f4 with result from task 456
parsl.dataflow.dflow DEBUG: Task 456 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 491a8dbc1bed344780f8109dc4f1b0ee
parsl.dataflow.memoization INFO: Task 457 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 648
parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5ba726dc-e9fa-4415-b39b-d4faab341260_characterizeImage_946180_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5ba726dc-e9fa-4415-b39b-d4faab341260_characterizeImage_946180_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/00547a90-29f3-4f02-b9e2-95ccfae2cb46_isr_946180_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/00547a90-29f3-4f02-b9e2-95ccfae2cb46_isr_946180_54.stderr
parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52fe1fbca6279907a3375bbab087419e with result from task 465
parsl.dataflow.dflow DEBUG: Task 465 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 22d73add482284b32267e1ccc73f90d8
parsl.dataflow.memoization INFO: Task 466 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 649
parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/171dac64-dfdd-4fb9-a233-edfd4fc4aedb_characterizeImage_946072_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/171dac64-dfdd-4fb9-a233-edfd4fc4aedb_characterizeImage_946072_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/12daaafb-7208-4f82-9663-245a9a1e274d_isr_946072_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/12daaafb-7208-4f82-9663-245a9a1e274d_isr_946072_24.stderr
parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef35022fa38c2c6535ca92cb0be622b8 with result from task 462
parsl.dataflow.dflow DEBUG: Task 462 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 5de8fc63606772f3e2a78ee9d04ed46d
parsl.dataflow.memoization INFO: Task 463 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 650
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3e23d4b3-954b-47cb-a762-20fb612548c2_characterizeImage_946181_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3e23d4b3-954b-47cb-a762-20fb612548c2_characterizeImage_946181_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d08dc57b-ae49-46e1-91d5-9f903f0c0424_isr_946181_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/d08dc57b-ae49-46e1-91d5-9f903f0c0424_isr_946181_57.stderr
parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47716c856664f36685df30d7cfbcc316 with result from task 474
parsl.dataflow.dflow DEBUG: Task 474 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 83daf6301e76f1d95372dc8ac5f95565
parsl.dataflow.memoization INFO: Task 475 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 651
parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ba8e5fd3-0a67-4e84-8523-736d7b030f4f_characterizeImage_946074_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ba8e5fd3-0a67-4e84-8523-736d7b030f4f_characterizeImage_946074_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4a46f180-5752-47c7-97a4-2883a7fea544_isr_946074_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/4a46f180-5752-47c7-97a4-2883a7fea544_isr_946074_46.stderr
parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 83d6d4d073e072bf776906b1fcc0f7a4 with result from task 492
parsl.dataflow.dflow DEBUG: Task 492 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 1a02357d7046e4466995ec91826b0f60
parsl.dataflow.memoization INFO: Task 493 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 652
parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5ea8cb2b-d830-454f-9282-39d125136784_characterizeImage_946074_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5ea8cb2b-d830-454f-9282-39d125136784_characterizeImage_946074_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/3fd9931a-cf86-4779-ae42-8f9b9013dfc4_isr_946074_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/3fd9931a-cf86-4779-ae42-8f9b9013dfc4_isr_946074_51.stderr
parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc3141fac5d4217b622c9277fc9b8e2c with result from task 489
parsl.dataflow.dflow DEBUG: Task 489 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 728df41a0f1721fc89c754ccb3fb7567
parsl.dataflow.memoization INFO: Task 490 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 653
parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0d45ff8a-1a35-4f2c-ab94-01cbc5cc4d22_characterizeImage_946181_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0d45ff8a-1a35-4f2c-ab94-01cbc5cc4d22_characterizeImage_946181_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/742ed215-3d88-4b43-bbd7-6780e63b3963_isr_946181_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/742ed215-3d88-4b43-bbd7-6780e63b3963_isr_946181_29.stderr
parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07a8559d17f5a86fcdb1c3caad1ff3d0 with result from task 483
parsl.dataflow.dflow DEBUG: Task 483 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 484 has memoization hash f64296fb39e4e17871b0dd56df9f6c70
parsl.dataflow.memoization INFO: Task 484 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 654
parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/481e3cbf-6ff1-432a-9487-eb5e3d988c10_characterizeImage_946073_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/481e3cbf-6ff1-432a-9487-eb5e3d988c10_characterizeImage_946073_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/757b25ce-f0c5-40a9-9eca-d17cdc708e2d_isr_946073_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/757b25ce-f0c5-40a9-9eca-d17cdc708e2d_isr_946073_27.stderr
parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 955621d4fe91b9d31cd33c8803a8137f with result from task 447
parsl.dataflow.dflow DEBUG: Task 447 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 799cb18df949caab270841927a00ae25
parsl.dataflow.memoization INFO: Task 448 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 655
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4deeba74-f8d5-4f2e-b395-e285b5ec1950_characterizeImage_946072_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4deeba74-f8d5-4f2e-b395-e285b5ec1950_characterizeImage_946072_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3bb30c9-4811-44ac-a0de-184ca981fdaf_isr_946072_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/c3bb30c9-4811-44ac-a0de-184ca981fdaf_isr_946072_49.stderr
parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63b8e5e2aa1b8d79b2fe57eb67626bf3 with result from task 459
parsl.dataflow.dflow DEBUG: Task 459 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 460 has memoization hash fe3decec0c9649cabe2a4c74f55e9d86
parsl.dataflow.memoization INFO: Task 460 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:24 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 656
parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5f5f8fbd-95e4-41fe-896f-7f9a54da4fc9_characterizeImage_946182_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5f5f8fbd-95e4-41fe-896f-7f9a54da4fc9_characterizeImage_946182_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/138cac86-975a-497a-a3aa-f8d77c673783_isr_946182_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/138cac86-975a-497a-a3aa-f8d77c673783_isr_946182_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:24 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:25 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:25 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84f3daea523138cf611a7b6989ca21fe with result from task 352
parsl.dataflow.dflow DEBUG: Task 352 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 353 has memoization hash e47398088d0f2668dc6dc5774780657e
parsl.dataflow.memoization INFO: Task 353 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 657
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ace4bded-84af-4157-8d2c-c3d2db56ecca_calibrate_946072_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ace4bded-84af-4157-8d2c-c3d2db56ecca_calibrate_946072_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0ba8fa28-f898-475d-a543-661a5d75b02a_characterizeImage_946072_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0ba8fa28-f898-475d-a543-661a5d75b02a_characterizeImage_946072_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:26 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 409 try 0 failed
parsl.dataflow.dflow INFO: Task 409 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 42d77eb14a3a84b3544107491ef2a6fb
parsl.dataflow.memoization INFO: Task 409 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 409 try 1 launched on executor multi with executor id 658
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:26 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aca6a23fc6a7c7b7080a4256f092e56d with result from task 721
parsl.dataflow.dflow DEBUG: Task 721 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 2798d6854fb236550087989817ae51f2
parsl.dataflow.memoization INFO: Task 722 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 659
parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93b22042-7099-4672-9117-69562dfb75e5_calibrate_946073_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93b22042-7099-4672-9117-69562dfb75e5_calibrate_946073_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ac79f36c-7389-4fe3-9f9e-d1ba2c3077d2_characterizeImage_946073_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ac79f36c-7389-4fe3-9f9e-d1ba2c3077d2_characterizeImage_946073_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:26 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:27 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:27 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bef71a53466719da3ce395ba07557b16 with result from task 477
parsl.dataflow.dflow DEBUG: Task 477 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 478 has memoization hash c3d5517014a3934b7e0fdb215a2120d9
parsl.dataflow.memoization INFO: Task 478 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 660
parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c4a33271-a563-4db0-ad0f-509a2c294e27_characterizeImage_946072_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c4a33271-a563-4db0-ad0f-509a2c294e27_characterizeImage_946072_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/70de7056-1273-4908-94e8-b94d48e36b35_isr_946072_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/70de7056-1273-4908-94e8-b94d48e36b35_isr_946072_48.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37a11fa9a733935adc5619fb8b5ae5f9 with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash e632e1391866b007af6e055510410d1f
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 661
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/478c8662-1163-4f4d-b66f-b6531e47f205_calibrate_946072_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/478c8662-1163-4f4d-b66f-b6531e47f205_calibrate_946072_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/47cb0f5a-d920-4722-85ed-bb545b5de958_characterizeImage_946072_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/47cb0f5a-d920-4722-85ed-bb545b5de958_characterizeImage_946072_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:28 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 610 try 0 failed
parsl.dataflow.dflow INFO: Task 610 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 610 has memoization hash c72e7ce086c29ce87447fb99659d9d88
parsl.dataflow.memoization INFO: Task 610 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 610 try 1 launched on executor multi with executor id 662
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4cfbd906875fd4e8fbbc33a5126e031 with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash bfc73f0c6e69360c8c9189ceff56ced2
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 663
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/83cbc48e-9cc4-493e-a529-f448f814ffc4_calibrate_946072_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/83cbc48e-9cc4-493e-a529-f448f814ffc4_calibrate_946072_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0d0973e1-c8ab-4062-b0a9-282092513c81_characterizeImage_946072_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0d0973e1-c8ab-4062-b0a9-282092513c81_characterizeImage_946072_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:28 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:28 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:29 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40f019b89443af7b4070cf121dcf7718 with result from task 202
parsl.dataflow.dflow DEBUG: Task 202 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 203 has memoization hash ff72189743838f7538fe44a88842df55
parsl.dataflow.memoization INFO: Task 203 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 664
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/289546f5-5586-453b-b1c2-2b1a6ece237c_calibrate_946073_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/289546f5-5586-453b-b1c2-2b1a6ece237c_calibrate_946073_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ad93d17e-4dc2-4285-a633-1be431e45509_characterizeImage_946073_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ad93d17e-4dc2-4285-a633-1be431e45509_characterizeImage_946073_50.stderr
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 48f5e8d3cfe003003e84673802be5039 with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 7df78bc8dd831e37dce203e10ed4954f
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 665
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4bbc053c-b961-4531-96f2-19aa8d7932fb_calibrate_946073_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4bbc053c-b961-4531-96f2-19aa8d7932fb_calibrate_946073_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/68ce8470-b918-4ce9-96da-a2c5bbf083d1_characterizeImage_946073_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/68ce8470-b918-4ce9-96da-a2c5bbf083d1_characterizeImage_946073_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:29 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:30 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:30 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a86c60c535148267711606d5c02a83c1 with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 9d76f60a0fbfb85a2e98ba8fd3467e1f
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 666
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2472aa6a-a767-4549-b9ac-f6d4c9f91a0e_calibrate_946072_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2472aa6a-a767-4549-b9ac-f6d4c9f91a0e_calibrate_946072_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/35384805-789c-45d7-8505-1dc6a1490223_characterizeImage_946072_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/35384805-789c-45d7-8505-1dc6a1490223_characterizeImage_946072_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:30 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:31 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:31 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:32 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:32 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d649b0637dee0ab81535c416bdd40a72 with result from task 379
parsl.dataflow.dflow DEBUG: Task 379 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 37db362567fda8c3f87c408c4f54bdd6
parsl.dataflow.memoization INFO: Task 380 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 667
parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/97c1f05f-154a-4767-bab4-0c69fe695ccd_calibrate_946073_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/97c1f05f-154a-4767-bab4-0c69fe695ccd_calibrate_946073_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecd2800c-8c72-4bf9-a412-0f16bdf19594_characterizeImage_946073_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecd2800c-8c72-4bf9-a412-0f16bdf19594_characterizeImage_946073_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:32 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 278ac09c1d36596effc59ea2be5831d4 with result from task 820
parsl.dataflow.dflow DEBUG: Task 820 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 575c7fa8bf19a3ff8dcc434d4ad807cc
parsl.dataflow.memoization INFO: Task 821 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 668
parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c1a1be0b-0fca-4cb3-a5d4-5fdaa72f8446_calibrate_946074_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c1a1be0b-0fca-4cb3-a5d4-5fdaa72f8446_calibrate_946074_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e3cfbf29-b437-4d26-a5b3-60086814f200_characterizeImage_946074_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e3cfbf29-b437-4d26-a5b3-60086814f200_characterizeImage_946074_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:33 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:33 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:34 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5dee24e21d5685cdf661fe54495c7e6 with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash fa41f060355feec3f4b7b5a8b9ceff05
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 669
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3ebee171-5337-4db1-a31b-72938b7e3e28_calibrate_946073_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3ebee171-5337-4db1-a31b-72938b7e3e28_calibrate_946073_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7cb957a4-40ad-4286-8b4f-3976f1c98080_characterizeImage_946073_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7cb957a4-40ad-4286-8b4f-3976f1c98080_characterizeImage_946073_10.stderr
parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9f127bd8cdaf1c3f3a009eef3d99ee4d with result from task 211
parsl.dataflow.dflow DEBUG: Task 211 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 68419a9226602c1cca1524dc92c70942
parsl.dataflow.memoization INFO: Task 212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 670
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c6fc4235-33a2-4e52-b278-a2c8e1d04454_calibrate_946072_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c6fc4235-33a2-4e52-b278-a2c8e1d04454_calibrate_946072_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/34cc8df6-7576-45a0-9901-b0952ab91cc4_characterizeImage_946072_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/34cc8df6-7576-45a0-9901-b0952ab91cc4_characterizeImage_946072_15.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 430 try 1 failed
parsl.dataflow.dflow ERROR: Task 430 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 574cab65289e73b66f21d667fed0a3e9 with result from task 430
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 431 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 431 try 0 failed
parsl.dataflow.dflow INFO: Task 431 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/881ebba7-dfe3-4af5-b301-f700d899ba71_calibrate_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/881ebba7-dfe3-4af5-b301-f700d899ba71_calibrate_946181_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/99316205-fcb1-4b5c-9012-d7b937b697c4_characterizeImage_946181_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:34 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8737ee17b1dabdaee0935ce78d31e9b with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 39cbab66dd4cc9d7c8b634f75bed6b45
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 671
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a3bc34f7-97ec-4c11-be45-dcfeccafd3a3_calibrate_946073_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a3bc34f7-97ec-4c11-be45-dcfeccafd3a3_calibrate_946073_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9e39a269-89af-42d9-82b6-df35c55193ba_characterizeImage_946073_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9e39a269-89af-42d9-82b6-df35c55193ba_characterizeImage_946073_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:35 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:35 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:35 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:36 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:36 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c94026adf5cf286f35831d5b30d7b87e with result from task 912
parsl.dataflow.dflow DEBUG: Task 912 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 23b1ccbda8c45f600b00cf469cccc246
parsl.dataflow.memoization INFO: Task 913 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 672
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/bf812fdb-66b9-44b9-b876-16bb6c81f2dc_characterizeImage_946073_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/bf812fdb-66b9-44b9-b876-16bb6c81f2dc_characterizeImage_946073_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80443b3f-f3f6-4ced-8ca9-ce020747df10_isr_946073_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/80443b3f-f3f6-4ced-8ca9-ce020747df10_isr_946073_35.stderr
parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64ccfa54a29425147d61c99a6e589991 with result from task 924
parsl.dataflow.dflow DEBUG: Task 924 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 4af90318d69a9a576b0a4478bc8b4d95
parsl.dataflow.memoization INFO: Task 925 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 673
parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/94223191-96fd-4ab5-92c7-853383b7070c_characterizeImage_946073_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/94223191-96fd-4ab5-92c7-853383b7070c_characterizeImage_946073_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/41097f3e-372b-48d6-a088-cbf8e0e26fe2_isr_946073_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/41097f3e-372b-48d6-a088-cbf8e0e26fe2_isr_946073_43.stderr
parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e17265a2c19206d923f63a154b6bc3f with result from task 921
parsl.dataflow.dflow DEBUG: Task 921 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 0869b5e036eb4233ccffbff147286666
parsl.dataflow.memoization INFO: Task 922 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 674
parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e96b017d-ad8a-4d87-a8c5-d4b83492789a_characterizeImage_946181_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e96b017d-ad8a-4d87-a8c5-d4b83492789a_characterizeImage_946181_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b5c02d06-0bda-4661-b639-9f79bac8b36b_isr_946181_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b5c02d06-0bda-4661-b639-9f79bac8b36b_isr_946181_62.stderr
parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0c31679165e5a6dc33c1bb24d3f335d with result from task 903
parsl.dataflow.dflow DEBUG: Task 903 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 904 has memoization hash ff963189d0c194a5126324be7f1c155e
parsl.dataflow.memoization INFO: Task 904 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 675
parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3fc808d7-3a6d-46df-8f21-415d25874bba_characterizeImage_946074_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3fc808d7-3a6d-46df-8f21-415d25874bba_characterizeImage_946074_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ad8603f4-3ae2-497d-a5be-375b6a256a75_isr_946074_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ad8603f4-3ae2-497d-a5be-375b6a256a75_isr_946074_12.stderr
parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6da41c8e810ac5f22dc464fd28c75bcb with result from task 885
parsl.dataflow.dflow DEBUG: Task 885 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 846e1941fc71b274da7ecc949b5c262a
parsl.dataflow.memoization INFO: Task 886 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 676
parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6dd8b374-037c-47b7-ae85-1c91a7974fb5_characterizeImage_946181_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6dd8b374-037c-47b7-ae85-1c91a7974fb5_characterizeImage_946181_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/70640ead-f4d8-4914-83aa-560da32985ba_isr_946181_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/70640ead-f4d8-4914-83aa-560da32985ba_isr_946181_48.stderr
parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd18276aeafa4a87929f49fbd33a2d24 with result from task 891
parsl.dataflow.dflow DEBUG: Task 891 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 9767f240982f26b1641dcb307bcb58ad
parsl.dataflow.memoization INFO: Task 892 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 677
parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/eb303985-2bc0-4788-960c-f872b6187bc3_characterizeImage_946073_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/eb303985-2bc0-4788-960c-f872b6187bc3_characterizeImage_946073_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/521cc015-9db5-4074-a9b6-41268a09dac7_isr_946073_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/521cc015-9db5-4074-a9b6-41268a09dac7_isr_946073_52.stderr
parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98726af768eca148ee683323ae3f2329 with result from task 894
parsl.dataflow.dflow DEBUG: Task 894 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 895 has memoization hash dc25420bdeb558916cfd7ecfccdc8795
parsl.dataflow.memoization INFO: Task 895 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 678
parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/11274884-ee40-4aa9-9475-f871aaf39c56_characterizeImage_946072_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/11274884-ee40-4aa9-9475-f871aaf39c56_characterizeImage_946072_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bc335be7-7cf0-4578-8824-85eb9af26292_isr_946072_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/bc335be7-7cf0-4578-8824-85eb9af26292_isr_946072_31.stderr
parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebb10ee459864ce451542a7ed0d453af with result from task 900
parsl.dataflow.dflow DEBUG: Task 900 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 901 has memoization hash b907664398dc68b3455caf4ed1eeed7d
parsl.dataflow.memoization INFO: Task 901 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 679
parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/435feb3a-29c8-4937-82d4-891cf2704a47_characterizeImage_946072_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/435feb3a-29c8-4937-82d4-891cf2704a47_characterizeImage_946072_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fb3fe6c-6eb4-4953-a671-2f27340c1bcb_isr_946072_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/8fb3fe6c-6eb4-4953-a671-2f27340c1bcb_isr_946072_18.stderr
parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3086f9ccb6809614c3cdbb2a5f581e3 with result from task 909
parsl.dataflow.dflow DEBUG: Task 909 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 910 has memoization hash fd19cbfdeac1afddecabc2d93f1d8bbb
parsl.dataflow.memoization INFO: Task 910 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 680
parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5cae9711-b02e-4b09-aa52-42ce564dea92_characterizeImage_946182_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5cae9711-b02e-4b09-aa52-42ce564dea92_characterizeImage_946182_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9b51743c-aea5-4e7f-9d5a-8fe50d32ca76_isr_946182_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9b51743c-aea5-4e7f-9d5a-8fe50d32ca76_isr_946182_49.stderr
parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2a7baaa6e1d9ee232363695d96fd382 with result from task 927
parsl.dataflow.dflow DEBUG: Task 927 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 512180684046d13be1f25a67489c2862
parsl.dataflow.memoization INFO: Task 928 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 681
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5e2a5a70-a3b3-46dd-9205-4fd133d36bcc_characterizeImage_946074_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5e2a5a70-a3b3-46dd-9205-4fd133d36bcc_characterizeImage_946074_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9914b711-e456-44b4-82f3-4f57fd8c70dd_isr_946074_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9914b711-e456-44b4-82f3-4f57fd8c70dd_isr_946074_13.stderr
parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7748f6a7d9dbaaadc056a6a4d9bc4a8 with result from task 930
parsl.dataflow.dflow DEBUG: Task 930 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 931 has memoization hash cb9fb2838f3e8dc44c98e5a6edbcb8d0
parsl.dataflow.memoization INFO: Task 931 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 682
parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/0095620b-6bb3-42df-9560-652f25095933_characterizeImage_946073_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/0095620b-6bb3-42df-9560-652f25095933_characterizeImage_946073_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6ac0a71a-2e07-47a2-b5e0-1289a8a0c97a_isr_946073_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/6ac0a71a-2e07-47a2-b5e0-1289a8a0c97a_isr_946073_53.stderr
parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3ef31dcf5aa2308c47f28e02f62635b with result from task 915
parsl.dataflow.dflow DEBUG: Task 915 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 916 has memoization hash c68a257cb11de08b9c71d80608eb5482
parsl.dataflow.memoization INFO: Task 916 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 683
parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d9a088b4-3821-47d6-9af8-0e2586f10546_characterizeImage_946181_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d9a088b4-3821-47d6-9af8-0e2586f10546_characterizeImage_946181_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c89dea4c-786e-4069-9012-2196fcba5326_isr_946181_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/c89dea4c-786e-4069-9012-2196fcba5326_isr_946181_9.stderr
parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 992af664f45eb6b4eeebc6d742d0a208 with result from task 888
parsl.dataflow.dflow DEBUG: Task 888 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 6dbf26e675fb703d35e59253ba826cd3
parsl.dataflow.memoization INFO: Task 889 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 684
parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/e991d81a-389f-4c09-a142-23a811a7aa8f_characterizeImage_946182_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/e991d81a-389f-4c09-a142-23a811a7aa8f_characterizeImage_946182_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/338e95fe-9473-4a21-a182-643c20e70561_isr_946182_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/338e95fe-9473-4a21-a182-643c20e70561_isr_946182_42.stderr
parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50e91d2ee4dc5b0bbcee4ae71d8a1fc9 with result from task 906
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:37 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow DEBUG: Task 906 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 84b1bd03b1aeae5dc5b464932e26e7fa
parsl.dataflow.memoization INFO: Task 907 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 685
parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/57bac90d-b02b-43a4-b4d1-1094ad6acfd1_characterizeImage_946182_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/57bac90d-b02b-43a4-b4d1-1094ad6acfd1_characterizeImage_946182_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/780b6721-9fa1-4bf6-bcd0-55511acd6ec0_isr_946182_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/780b6721-9fa1-4bf6-bcd0-55511acd6ec0_isr_946182_46.stderr
parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7b7a2959324f19bedc589c65de08cba with result from task 918
parsl.dataflow.dflow DEBUG: Task 918 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 919 has memoization hash c2ffe02c4baf6938a35e3c05b9eefb31
parsl.dataflow.memoization INFO: Task 919 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 686
parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3460f65e-6846-4277-9479-30cdcca10a8f_characterizeImage_946074_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3460f65e-6846-4277-9479-30cdcca10a8f_characterizeImage_946074_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/688b831a-0a77-4777-8855-c384335162cf_isr_946074_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/688b831a-0a77-4777-8855-c384335162cf_isr_946074_19.stderr
parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de6d512ef872c70164cdfe75e581c568 with result from task 897
parsl.dataflow.dflow DEBUG: Task 897 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 328b83c9e04ef73a999ceca91dc3312a
parsl.dataflow.memoization INFO: Task 898 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 687
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ca58d1d-8ce7-4331-837b-2356ae6b53f4_characterizeImage_946074_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ca58d1d-8ce7-4331-837b-2356ae6b53f4_characterizeImage_946074_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9f63037c-fb51-49f5-a754-260e748a8eb2_isr_946074_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9f63037c-fb51-49f5-a754-260e748a8eb2_isr_946074_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:37 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f11a3fcb9c4ff6bbd45b7bd693c1dec6 with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash e654fa0201f731e7c6e71a3bd70e1542
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 688
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e2abe424-a741-4ca0-884c-f6b59171c422_calibrate_946072_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e2abe424-a741-4ca0-884c-f6b59171c422_calibrate_946072_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8843b2ff-275b-45f7-b6f8-bf9b8290cf05_characterizeImage_946072_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8843b2ff-275b-45f7-b6f8-bf9b8290cf05_characterizeImage_946072_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:38 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8de02a3a3589ba28a1368bfc4576c5cf with result from task 657
parsl.dataflow.dflow DEBUG: Task 657 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 42b21c639cca561a201179450367845f
parsl.dataflow.memoization INFO: Task 658 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 689
parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d44b3d8b-263f-474c-a366-d3f0328e0530_characterizeImage_946180_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d44b3d8b-263f-474c-a366-d3f0328e0530_characterizeImage_946180_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a2c01222-6a7c-42d0-9b5d-255c8495a630_isr_946180_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/a2c01222-6a7c-42d0-9b5d-255c8495a630_isr_946180_40.stderr
parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf90a58e215005e43870e4396025195b with result from task 648
parsl.dataflow.dflow DEBUG: Task 648 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 302a71c2a2e3a648fa6959516286fd14
parsl.dataflow.memoization INFO: Task 649 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088efb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 690
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/fc5869a3-9f3e-4bc5-b03e-08cce6d89c24_characterizeImage_946072_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/fc5869a3-9f3e-4bc5-b03e-08cce6d89c24_characterizeImage_946072_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/23e248f1-511f-41e3-bee7-c8c493e4ed95_isr_946072_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/23e248f1-511f-41e3-bee7-c8c493e4ed95_isr_946072_1.stderr
parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09a9f10336834680afc120fba473b8a3 with result from task 645
parsl.dataflow.dflow DEBUG: Task 645 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 79cc59cc806aef1b64d6b5c6121b2d54
parsl.dataflow.memoization INFO: Task 646 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 691
parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/d8ebca9a-2b0e-46b6-8fc3-b6627e8fd547_characterizeImage_946182_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/d8ebca9a-2b0e-46b6-8fc3-b6627e8fd547_characterizeImage_946182_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9cc70ad5-bf38-4758-b160-78b6fcca41ce_isr_946182_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/9cc70ad5-bf38-4758-b160-78b6fcca41ce_isr_946182_32.stderr
parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1e448b14f9730ac60216153b689e34 with result from task 642
parsl.dataflow.dflow DEBUG: Task 642 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 0c94095be96a68f47f77c48f0512d673
parsl.dataflow.memoization INFO: Task 643 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 692
parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e806b8e4-bcac-4be2-b202-89d458c64d00_characterizeImage_946073_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e806b8e4-bcac-4be2-b202-89d458c64d00_characterizeImage_946073_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/609434ca-8ddd-4d6f-b373-e18db27e1923_isr_946073_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/609434ca-8ddd-4d6f-b373-e18db27e1923_isr_946073_29.stderr
parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e55de0c29287e09958c035a881982aee with result from task 639
parsl.dataflow.dflow DEBUG: Task 639 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 099f779491164e98481852bb75167152
parsl.dataflow.memoization INFO: Task 640 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 693
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/25f521cd-ea68-49ed-8d38-0baf9053bea9_characterizeImage_946182_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/25f521cd-ea68-49ed-8d38-0baf9053bea9_characterizeImage_946182_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a35dd479-e487-4c39-a391-1234c20ebab2_isr_946182_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/a35dd479-e487-4c39-a391-1234c20ebab2_isr_946182_12.stderr
parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f55910e9817beba58298d326c387890 with result from task 654
parsl.dataflow.dflow DEBUG: Task 654 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 655 has memoization hash da939469ad61317cad5e2e70e31994b9
parsl.dataflow.memoization INFO: Task 655 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 694
parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e6f803a0-0148-4275-91c3-36a3261d3d75_characterizeImage_946180_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e6f803a0-0148-4275-91c3-36a3261d3d75_characterizeImage_946180_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b164f24-c574-4dc8-88bb-df21f6a67e8b_isr_946180_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/2b164f24-c574-4dc8-88bb-df21f6a67e8b_isr_946180_21.stderr
parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc1165301391547fcfe1be26b225581a with result from task 651
parsl.dataflow.dflow DEBUG: Task 651 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 652 has memoization hash eac5a9b68036748a2c29e95dee8acfec
parsl.dataflow.memoization INFO: Task 652 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088efe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 695
parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d8d53ad7-8a9b-4d97-bc6f-e7e4b51f75b9_characterizeImage_946074_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d8d53ad7-8a9b-4d97-bc6f-e7e4b51f75b9_characterizeImage_946074_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/79f3f393-2cf0-4267-a1cf-8713afb81963_isr_946074_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/79f3f393-2cf0-4267-a1cf-8713afb81963_isr_946074_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:38 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 369ba38c1db95e03e2f408e6ce4986e8 with result from task 660
parsl.dataflow.dflow DEBUG: Task 660 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 960ea0a656f948baacb279eb72dec693
parsl.dataflow.memoization INFO: Task 661 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 696
parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c64f463b-d22a-4f4e-ba2e-f79b1056ffa0_characterizeImage_946073_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c64f463b-d22a-4f4e-ba2e-f79b1056ffa0_characterizeImage_946073_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/fd4ffa07-dfc7-4d6a-b3bc-ef71a100b94b_isr_946073_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/fd4ffa07-dfc7-4d6a-b3bc-ef71a100b94b_isr_946073_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:39 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:39 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:39 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a946235fb0ad5441ce084935609e3293 with result from task 388
parsl.dataflow.dflow DEBUG: Task 388 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 389 has memoization hash cd2a5280f254db98b87fbd75d9970e24
parsl.dataflow.memoization INFO: Task 389 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 697
parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2730465a-04d0-4a70-8dd5-4af2a5eb6d5c_calibrate_946072_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2730465a-04d0-4a70-8dd5-4af2a5eb6d5c_calibrate_946072_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/07b287b7-c36b-4d22-9c6c-94a2b53ff981_characterizeImage_946072_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/07b287b7-c36b-4d22-9c6c-94a2b53ff981_characterizeImage_946072_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:40 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53ef5333dc70d4429b905e01603177a4 with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 7b4ec23b99408ca8e6132d083245c599
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 698
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34056ee1-e1a1-436d-9980-601a406ef2e8_calibrate_946181_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34056ee1-e1a1-436d-9980-601a406ef2e8_calibrate_946181_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/97de818e-7075-4d8f-94a7-bc08342edb91_characterizeImage_946181_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/97de818e-7075-4d8f-94a7-bc08342edb91_characterizeImage_946181_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:40 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:41 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0c89228bcc3126a9efe85f27f974d35 with result from task 283
parsl.dataflow.dflow DEBUG: Task 283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 6cc4c05f30028f4866c7ab665fe1e495
parsl.dataflow.memoization INFO: Task 284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 699
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/cb021e2d-a4ef-4c41-973e-763d68eb8228_calibrate_946072_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/cb021e2d-a4ef-4c41-973e-763d68eb8228_calibrate_946072_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/adbe60e2-97b7-4276-8594-cd65e46a649e_characterizeImage_946072_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/adbe60e2-97b7-4276-8594-cd65e46a649e_characterizeImage_946072_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:41 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:41 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:42 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:42 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:43 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 667ab15f04f1ba456465b3436d56caa3 with result from task 268
parsl.dataflow.dflow DEBUG: Task 268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 52db26d99fe15d08cb0670234b48bcf5
parsl.dataflow.memoization INFO: Task 269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 700
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/87d1092b-ff25-401d-8213-3d809fc7e0e1_calibrate_946073_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/87d1092b-ff25-401d-8213-3d809fc7e0e1_calibrate_946073_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4c593004-ef7b-4aa2-b26e-e2b03f8df216_characterizeImage_946073_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4c593004-ef7b-4aa2-b26e-e2b03f8df216_characterizeImage_946073_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:43 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:44 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8430c6161fb63ce7bc40c451a0de2583 with result from task 370
parsl.dataflow.dflow DEBUG: Task 370 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 596a8c3a5903b708f1a2c2239ae46793
parsl.dataflow.memoization INFO: Task 371 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 701
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/9c491b54-a9ec-43c0-b661-39824d07a910_calibrate_946074_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/9c491b54-a9ec-43c0-b661-39824d07a910_calibrate_946074_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/22281e2b-fcf5-47ef-a400-caec837424af_characterizeImage_946074_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/22281e2b-fcf5-47ef-a400-caec837424af_characterizeImage_946074_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:44 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:44 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89aaa57a3f36bbad0d0a33a3b859baf7 with result from task 403
parsl.dataflow.dflow DEBUG: Task 403 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 8242cff426e524754e3d225c488f1531
parsl.dataflow.memoization INFO: Task 404 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 702
parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0db56701-0a64-4eb0-887f-894833ff87e8_calibrate_946074_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0db56701-0a64-4eb0-887f-894833ff87e8_calibrate_946074_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/1709020b-7329-4630-84ad-4956984764f4_characterizeImage_946074_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/1709020b-7329-4630-84ad-4956984764f4_characterizeImage_946074_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:45 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eecca9237bcc77c2aaba7e94632a19f0 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash a360d24acadd11b782aae373cc80a9fe
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 703
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/86b9dd13-be38-4ca8-9acc-52d4f5e1799c_calibrate_946074_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/86b9dd13-be38-4ca8-9acc-52d4f5e1799c_calibrate_946074_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/46bc9375-fee6-4038-b442-a5679437d612_characterizeImage_946074_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/46bc9375-fee6-4038-b442-a5679437d612_characterizeImage_946074_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:45 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:46 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4867fab6974ffa7507e65d4e20e67d8e with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 22c3785f0f7c43a8c077a502adf0808c
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 704
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fee4ea8c-02fc-45ae-bd88-61b517d62c4f_calibrate_946074_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fee4ea8c-02fc-45ae-bd88-61b517d62c4f_calibrate_946074_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/4bf35adf-74ad-4eba-9291-862380eca317_characterizeImage_946074_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/4bf35adf-74ad-4eba-9291-862380eca317_characterizeImage_946074_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:46 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:47 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd9eaff85c2633d2b5374dade85a36e4 with result from task 1005
parsl.dataflow.dflow DEBUG: Task 1005 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash a0d001dc339f66360b97bdd9df1bd0ea
parsl.dataflow.memoization INFO: Task 1006 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 705
parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7efea9d2-c122-454d-8426-ad270c23ac9b_characterizeImage_946074_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7efea9d2-c122-454d-8426-ad270c23ac9b_characterizeImage_946074_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9154de2b-9329-4eaa-a5b6-52f4936dd99e_isr_946074_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9154de2b-9329-4eaa-a5b6-52f4936dd99e_isr_946074_25.stderr
parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9df5a8f701c642c5c386a471af4efe2e with result from task 1023
parsl.dataflow.dflow DEBUG: Task 1023 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash d78583f1500fc98cf98036eff9d2a88d
parsl.dataflow.memoization INFO: Task 1024 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 706
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/28a8320e-0daa-4e44-8f5d-005d2aa48522_isr_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/28a8320e-0daa-4e44-8f5d-005d2aa48522_isr_946180_2.stderr
parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d523da09d2e809016589529bdf72a9d4 with result from task 1014
parsl.dataflow.dflow DEBUG: Task 1014 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash b5215a8a4efec795ab490e10464d33f7
parsl.dataflow.memoization INFO: Task 1015 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 707
parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a91f84e7-bf7b-4b80-962f-350fee745422_characterizeImage_946182_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a91f84e7-bf7b-4b80-962f-350fee745422_characterizeImage_946182_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7a24098d-09b8-4e7a-a273-0a125d151b40_isr_946182_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/7a24098d-09b8-4e7a-a273-0a125d151b40_isr_946182_7.stderr
parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 390afc46801bbcde181a0c59becf9270 with result from task 1026
parsl.dataflow.dflow DEBUG: Task 1026 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash cb3d364cfebb00a56557da3b9e699c88
parsl.dataflow.memoization INFO: Task 1027 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 708
parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfc016be-9300-467c-9433-06e46804795b_characterizeImage_946182_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfc016be-9300-467c-9433-06e46804795b_characterizeImage_946182_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/51f58ccf-1a4c-4553-b562-85da0a1724a2_isr_946182_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/51f58ccf-1a4c-4553-b562-85da0a1724a2_isr_946182_37.stderr
parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5bcb30d11119b2e6036c369dc7b02a4e with result from task 1020
parsl.dataflow.dflow DEBUG: Task 1020 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 535be5ca0d45d6348d805c9a380f04f1
parsl.dataflow.memoization INFO: Task 1021 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:47 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 709
parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/7c22df4c-01fd-4908-8f74-a256378b4c40_characterizeImage_946072_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/7c22df4c-01fd-4908-8f74-a256378b4c40_characterizeImage_946072_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9065cdfa-dfcb-445b-922c-1504ed3b144d_isr_946072_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/9065cdfa-dfcb-445b-922c-1504ed3b144d_isr_946072_7.stderr
parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d60a699b0145e04d68832ca9e847f4df with result from task 1008
parsl.dataflow.dflow DEBUG: Task 1008 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 28843729de0cd32431569ff2ce5cd63e
parsl.dataflow.memoization INFO: Task 1009 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 710
parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0afc4433-d75d-43db-b08e-d974935d7158_characterizeImage_946180_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0afc4433-d75d-43db-b08e-d974935d7158_characterizeImage_946180_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/0bc7b2ab-2a01-418f-86e7-e42de1ed52a6_isr_946180_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/0bc7b2ab-2a01-418f-86e7-e42de1ed52a6_isr_946180_43.stderr
parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe60a149bb61aa976f84d7de79c7051 with result from task 1011
parsl.dataflow.dflow DEBUG: Task 1011 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash fc28f59c24bdff170937e98533c362c5
parsl.dataflow.memoization INFO: Task 1012 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 711
parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6c6b4150-e1ed-4cc6-9bfd-88bb4c6c0d7e_characterizeImage_946181_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6c6b4150-e1ed-4cc6-9bfd-88bb4c6c0d7e_characterizeImage_946181_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/12a418ab-7bd2-43a0-8ea9-75caa4adb229_isr_946181_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/12a418ab-7bd2-43a0-8ea9-75caa4adb229_isr_946181_7.stderr
parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aede78a8f9efa2f72a5438cca61dae03 with result from task 1017
parsl.dataflow.dflow DEBUG: Task 1017 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 8d6d51e4c5a9d3878758776c0d808580
parsl.dataflow.memoization INFO: Task 1018 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 712
parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2365734-2076-4518-833a-45f97b8c3245_characterizeImage_946073_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2365734-2076-4518-833a-45f97b8c3245_characterizeImage_946073_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9bc224c4-b271-425b-8e6b-e31cc40e1447_isr_946073_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9bc224c4-b271-425b-8e6b-e31cc40e1447_isr_946073_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:47 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 180edca4e1731bdc2a7c048843e0c992 with result from task 337
parsl.dataflow.dflow DEBUG: Task 337 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 1ec1a8f32ae4109eb6af8186e84ad169
parsl.dataflow.memoization INFO: Task 338 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 713
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d20d7954-b831-46ed-a981-8eecee73c91e_calibrate_946181_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d20d7954-b831-46ed-a981-8eecee73c91e_calibrate_946181_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50832a28-bcf2-4385-9b00-747bdb43231a_characterizeImage_946181_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50832a28-bcf2-4385-9b00-747bdb43231a_characterizeImage_946181_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:48 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74163c9ac7ae792e55fe74f3923c8ae5 with result from task 343
parsl.dataflow.dflow DEBUG: Task 343 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 5a692480244f35eade41d5b3083aacb1
parsl.dataflow.memoization INFO: Task 344 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 714
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d886fde2-5cda-45aa-84e3-0b3c3197e6fa_calibrate_946181_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d886fde2-5cda-45aa-84e3-0b3c3197e6fa_calibrate_946181_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/277842ee-4f53-4a87-867a-ddc20f13ffe9_characterizeImage_946181_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/277842ee-4f53-4a87-867a-ddc20f13ffe9_characterizeImage_946181_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:48 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:49 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd87cb22220b4d8ae008bca51ca83390 with result from task 984
parsl.dataflow.dflow DEBUG: Task 984 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 985 has memoization hash a2b1c1e1304570de9b6a0d3c9f1f45ad
parsl.dataflow.memoization INFO: Task 985 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 715
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7e51b755-51dc-45ab-a83f-92a56ad4743f_characterizeImage_946073_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7e51b755-51dc-45ab-a83f-92a56ad4743f_characterizeImage_946073_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ab5fb525-9d6c-40de-84d6-e3541a3bffe2_isr_946073_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/ab5fb525-9d6c-40de-84d6-e3541a3bffe2_isr_946073_25.stderr
parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc9b2b2da4c2b67d9b4229a7bf35416 with result from task 996
parsl.dataflow.dflow DEBUG: Task 996 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 997 has memoization hash f359b2a59ab391425f62b243c98ecf78
parsl.dataflow.memoization INFO: Task 997 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 716
parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b81ba4f0-4db3-42d6-9cd3-77eb55e108ee_characterizeImage_946182_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b81ba4f0-4db3-42d6-9cd3-77eb55e108ee_characterizeImage_946182_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/74099723-cf51-4888-b38b-055685e8faae_isr_946182_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/74099723-cf51-4888-b38b-055685e8faae_isr_946182_16.stderr
parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58e70d3edb843015f7b8b1eccdaeaa5c with result from task 981
parsl.dataflow.dflow DEBUG: Task 981 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 6309d088cc2c0b18d67daf57b0535738
parsl.dataflow.memoization INFO: Task 982 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 717
parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e1b08152-e56b-4ed7-913e-164bb11a71ec_characterizeImage_946073_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e1b08152-e56b-4ed7-913e-164bb11a71ec_characterizeImage_946073_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/43e7c4c5-5e3e-4253-bd9e-cd52fb430452_isr_946073_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/43e7c4c5-5e3e-4253-bd9e-cd52fb430452_isr_946073_57.stderr
parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c846c655610d2bef8604319f9e4198b with result from task 987
parsl.dataflow.dflow DEBUG: Task 987 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 988 has memoization hash f5e9564c21b43c6b90bbaa48cc52c749
parsl.dataflow.memoization INFO: Task 988 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 718
parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/545e7068-16ba-400c-a9e7-0445cc879021_characterizeImage_946074_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/545e7068-16ba-400c-a9e7-0445cc879021_characterizeImage_946074_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ff0ed01a-71c4-4bcb-8fc0-f0fc929a9869_isr_946074_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/ff0ed01a-71c4-4bcb-8fc0-f0fc929a9869_isr_946074_23.stderr
parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebc366254a414903ee238d4d0e025c9d with result from task 1002
parsl.dataflow.dflow DEBUG: Task 1002 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 99a10b0c45251efbe2a1c6066290c2bb
parsl.dataflow.memoization INFO: Task 1003 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 719
parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/55825a35-38c3-4d06-8c9e-bd7776f15092_characterizeImage_946074_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/55825a35-38c3-4d06-8c9e-bd7776f15092_characterizeImage_946074_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/935bb7b5-ba14-4948-8150-b576851b8cac_isr_946074_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/935bb7b5-ba14-4948-8150-b576851b8cac_isr_946074_52.stderr
parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa39d7421dcb246e606f78cdcf7721d6 with result from task 999
parsl.dataflow.dflow DEBUG: Task 999 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 3f8ae463e5cfef9f5fcd056c20d08a4c
parsl.dataflow.memoization INFO: Task 1000 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 720
parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4400bafb-7bac-460d-b68e-a1b5d0ad7670_characterizeImage_946073_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4400bafb-7bac-460d-b68e-a1b5d0ad7670_characterizeImage_946073_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8972a105-778c-4f14-b787-0c96941bfe4b_isr_946073_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/8972a105-778c-4f14-b787-0c96941bfe4b_isr_946073_26.stderr
parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78e5b79c02d2fbbd840102aa466f0230 with result from task 990
parsl.dataflow.dflow DEBUG: Task 990 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 95f2cab2908ff757b269ba6c1271bf58
parsl.dataflow.memoization INFO: Task 991 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 721
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/59f3c7b8-869e-4355-9643-7c4418b6bc4a_isr_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/59f3c7b8-869e-4355-9643-7c4418b6bc4a_isr_946072_61.stderr
parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4259c1b1e6595e7977b8ca8ac9df35 with result from task 993
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:49 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow DEBUG: Task 993 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 994 has memoization hash fa4160e84a8c5c8530e7a75bc02d8e17
parsl.dataflow.memoization INFO: Task 994 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 722
parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/14035c60-fa81-4cc9-85a0-673224eabe8f_characterizeImage_946073_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/14035c60-fa81-4cc9-85a0-673224eabe8f_characterizeImage_946073_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47328e9f-cada-430d-b345-2443ff9ac690_isr_946073_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/47328e9f-cada-430d-b345-2443ff9ac690_isr_946073_28.stderr
parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c21c39c0ac33c18893d9d20cdf0d6c47 with result from task 807
parsl.dataflow.dflow DEBUG: Task 807 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 808 has memoization hash e9ba0572c46a4fa1af12b5ce80ccaba0
parsl.dataflow.memoization INFO: Task 808 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 723
parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d68bb6e6-3a8b-4d13-96a2-7b5e1b7bc5f7_characterizeImage_946073_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d68bb6e6-3a8b-4d13-96a2-7b5e1b7bc5f7_characterizeImage_946073_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/62614a60-f1c5-490f-9efa-76c10957854e_isr_946073_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/62614a60-f1c5-490f-9efa-76c10957854e_isr_946073_42.stderr
parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db0401fdb0c5ab5f159faef5c65ebd21 with result from task 786
parsl.dataflow.dflow DEBUG: Task 786 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 5b074ebd771c10f32f54835f9fc7115b
parsl.dataflow.memoization INFO: Task 787 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 724
parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/2c62f3d0-f4b7-4b00-ac9b-4fa39d84b178_characterizeImage_946073_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/2c62f3d0-f4b7-4b00-ac9b-4fa39d84b178_characterizeImage_946073_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0858d0f1-8788-498a-bc3d-e88ccf6c9163_isr_946073_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0858d0f1-8788-498a-bc3d-e88ccf6c9163_isr_946073_33.stderr
parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13287307bf373c551e5f9e9301bc47db with result from task 792
parsl.dataflow.dflow DEBUG: Task 792 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 57c0f375fc1eb8fb4231986ff4a78be0
parsl.dataflow.memoization INFO: Task 793 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 725
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a3bb088-fa68-42f7-9564-c3f97a2a8f2e_characterizeImage_946073_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a3bb088-fa68-42f7-9564-c3f97a2a8f2e_characterizeImage_946073_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23de528a-7fb5-4b85-96e0-059139828192_isr_946073_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23de528a-7fb5-4b85-96e0-059139828192_isr_946073_40.stderr
parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b00f4c886bdc707ad21c1092a6427e35 with result from task 783
parsl.dataflow.dflow DEBUG: Task 783 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 358b3bd51dd34f16be921b776560ebac
parsl.dataflow.memoization INFO: Task 784 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 726
parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd5d9424-be8d-4d4d-8ac5-c3a75b32c9af_characterizeImage_946181_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd5d9424-be8d-4d4d-8ac5-c3a75b32c9af_characterizeImage_946181_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/442760e8-47e6-480f-8c4d-3d4c20cf9d2e_isr_946181_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/442760e8-47e6-480f-8c4d-3d4c20cf9d2e_isr_946181_59.stderr
parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37a90844a4b1a600c02ab7dfbfb0199f with result from task 804
parsl.dataflow.dflow DEBUG: Task 804 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 7418b30961e4ed6e1a15a265573eb166
parsl.dataflow.memoization INFO: Task 805 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 727
parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f770fc60-6862-41eb-bb98-a057f9606f5b_characterizeImage_946073_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f770fc60-6862-41eb-bb98-a057f9606f5b_characterizeImage_946073_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9dfac037-cc93-481e-be66-300019aba4c3_isr_946073_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/9dfac037-cc93-481e-be66-300019aba4c3_isr_946073_37.stderr
parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1aadc87a5ad40262ae240310e8292da3 with result from task 789
parsl.dataflow.dflow DEBUG: Task 789 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 7c067c1cb6ac1b8249459db9371b6d10
parsl.dataflow.memoization INFO: Task 790 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 728
parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/bf2da13f-fe1b-4b8e-ad6e-ea81ae374a5d_characterizeImage_946074_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/bf2da13f-fe1b-4b8e-ad6e-ea81ae374a5d_characterizeImage_946074_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9987bd13-97b4-4e56-a8b5-72d51256b2ea_isr_946074_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/9987bd13-97b4-4e56-a8b5-72d51256b2ea_isr_946074_15.stderr
parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19b298f603a99a1c209383397e813232 with result from task 798
parsl.dataflow.dflow DEBUG: Task 798 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 799 has memoization hash afd8ba9bb7f8571ca98842d1fc25fd88
parsl.dataflow.memoization INFO: Task 799 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 729
parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1aa29ad-d88f-4ddd-9706-497b4ab824a7_characterizeImage_946073_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1aa29ad-d88f-4ddd-9706-497b4ab824a7_characterizeImage_946073_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0332f59e-aad6-4fa2-9734-b1ee2932f0c2_isr_946073_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/0332f59e-aad6-4fa2-9734-b1ee2932f0c2_isr_946073_30.stderr
parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2bdebc7855a2e86382bb82b3899dacd4 with result from task 774
parsl.dataflow.dflow DEBUG: Task 774 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 336b7fd06c9707cc8558fb87da98a7ba
parsl.dataflow.memoization INFO: Task 775 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 730
parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/230bc3d2-a3ce-4cb5-914d-f6a79347ac3a_characterizeImage_946181_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/230bc3d2-a3ce-4cb5-914d-f6a79347ac3a_characterizeImage_946181_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/63b0f786-00f7-456a-8d62-41433795da49_isr_946181_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/63b0f786-00f7-456a-8d62-41433795da49_isr_946181_45.stderr
parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 433ba0f5dee72611eed2bbdd9a440cfa with result from task 765
parsl.dataflow.dflow DEBUG: Task 765 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 4d9e59571859a279d25d07527816dee0
parsl.dataflow.memoization INFO: Task 766 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 731
parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fe96cda2-d114-4cfe-b4b9-6457de059198_characterizeImage_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fe96cda2-d114-4cfe-b4b9-6457de059198_characterizeImage_946182_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/4ff20f97-6e57-49b7-9643-d37f637c77dc_isr_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/4ff20f97-6e57-49b7-9643-d37f637c77dc_isr_946182_31.stderr
parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aafa9eda03f333755f7aa55e0f75ddcd with result from task 795
parsl.dataflow.dflow DEBUG: Task 795 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 796 has memoization hash ee4cd158c1253740da39e5dc2b4162e6
parsl.dataflow.memoization INFO: Task 796 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 732
parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e6751cb4-f578-485f-8971-804449eace20_characterizeImage_946073_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e6751cb4-f578-485f-8971-804449eace20_characterizeImage_946073_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/63fd7a00-b843-4152-9241-8af230cfa323_isr_946073_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/63fd7a00-b843-4152-9241-8af230cfa323_isr_946073_55.stderr
parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9e8f54b31619552c61ec599a7095914 with result from task 762
parsl.dataflow.dflow DEBUG: Task 762 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 763 has memoization hash c5d16afd46c672edad939b76efbfdc64
parsl.dataflow.memoization INFO: Task 763 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 733
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1c316c6-307a-4100-b4bb-ba1de4a3c870_characterizeImage_946073_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1c316c6-307a-4100-b4bb-ba1de4a3c870_characterizeImage_946073_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/20838f3f-cd5a-4478-b5b7-2b31399e01df_isr_946073_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/20838f3f-cd5a-4478-b5b7-2b31399e01df_isr_946073_38.stderr
parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 246d6c3f2ebe7d9c83e2931bb617226d with result from task 771
parsl.dataflow.dflow DEBUG: Task 771 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 772 has memoization hash af1c38953920e812f7221613e8385bc9
parsl.dataflow.memoization INFO: Task 772 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 734
parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d4b80e25-25a5-4296-8316-ec79837b6643_characterizeImage_946073_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d4b80e25-25a5-4296-8316-ec79837b6643_characterizeImage_946073_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d5f13ff1-aeb6-4fb7-977f-5f082e629fbf_isr_946073_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d5f13ff1-aeb6-4fb7-977f-5f082e629fbf_isr_946073_58.stderr
parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7bf4fc18247a96671b1f6b4d075e86f with result from task 780
parsl.dataflow.dflow DEBUG: Task 780 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 781 has memoization hash cf63a538d96aa2bb64de4a9d63a7b0ce
parsl.dataflow.memoization INFO: Task 781 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 735
parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c5b02300-6c56-45cf-9fe6-7b85222f29c3_characterizeImage_946073_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c5b02300-6c56-45cf-9fe6-7b85222f29c3_characterizeImage_946073_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/93f5a6b4-2662-44b9-a711-6864b00c37c8_isr_946073_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/93f5a6b4-2662-44b9-a711-6864b00c37c8_isr_946073_46.stderr
parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f69b06116efd836ab85c64cf5da34f38 with result from task 801
parsl.dataflow.dflow DEBUG: Task 801 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 802 has memoization hash db8c5e475e54e9e701073bb8786a4ed6
parsl.dataflow.memoization INFO: Task 802 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 736
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ed26306a-a54b-4d75-b86f-2fa254d4ac5e_characterizeImage_946180_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ed26306a-a54b-4d75-b86f-2fa254d4ac5e_characterizeImage_946180_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7c955426-e7f7-427c-af98-37d9d79d69d9_isr_946180_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/7c955426-e7f7-427c-af98-37d9d79d69d9_isr_946180_13.stderr
parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea2c97b006e3ad63325cdd000b3a2f2d with result from task 768
parsl.dataflow.dflow DEBUG: Task 768 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 769 has memoization hash b5263371d2e9e913e55d31bc596bc274
parsl.dataflow.memoization INFO: Task 769 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 737
parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2b86150-4836-431b-93b4-cd75a2b65cdd_characterizeImage_946073_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2b86150-4836-431b-93b4-cd75a2b65cdd_characterizeImage_946073_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/36c820e3-95c0-47a8-87e6-3747b5370860_isr_946073_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/36c820e3-95c0-47a8-87e6-3747b5370860_isr_946073_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:50 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc4c9ba71f5927cae5d5779b23eabf97 with result from task 427
parsl.dataflow.dflow DEBUG: Task 427 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 428 has memoization hash a630aba8cf44189263fe98c73e26366e
parsl.dataflow.memoization INFO: Task 428 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 738
parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e723cfe2-ae88-4739-89f0-c7fab1aabc4f_calibrate_946074_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e723cfe2-ae88-4739-89f0-c7fab1aabc4f_calibrate_946074_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/692e0adf-8a7f-4bc7-a454-724d58ca2a15_characterizeImage_946074_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/692e0adf-8a7f-4bc7-a454-724d58ca2a15_characterizeImage_946074_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:50 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 634 try 0 failed
parsl.dataflow.dflow INFO: Task 634 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 8a99afa4eb9ed6ff4f0daaa813b24a03
parsl.dataflow.memoization INFO: Task 634 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 634 try 1 launched on executor multi with executor id 739
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:51 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:51 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:51 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a44f8dde81b2db89604e091ab0d59805 with result from task 861
parsl.dataflow.dflow DEBUG: Task 861 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 779c231920704f4c744697af55ff4984
parsl.dataflow.memoization INFO: Task 862 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088099e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 740
parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d50206d7-28b5-47a1-9667-0b3ef71d57e3_characterizeImage_946181_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d50206d7-28b5-47a1-9667-0b3ef71d57e3_characterizeImage_946181_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/09c617c6-9338-441d-b274-c42e6474ad64_isr_946181_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/09c617c6-9338-441d-b274-c42e6474ad64_isr_946181_23.stderr
parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry caccd52027c1c49fa4536f7e20514867 with result from task 867
parsl.dataflow.dflow DEBUG: Task 867 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 868 has memoization hash c2f4808eea9108188dbc77f37fa9a219
parsl.dataflow.memoization INFO: Task 868 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 741
parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1d77c8d2-1be8-4085-ace9-47dbdd6d8207_characterizeImage_946182_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1d77c8d2-1be8-4085-ace9-47dbdd6d8207_characterizeImage_946182_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/69473f3b-8e13-4557-92ac-816a759e1388_isr_946182_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/69473f3b-8e13-4557-92ac-816a759e1388_isr_946182_53.stderr
parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 210115e7f1bfb3a95a9b0c1f32281b1f with result from task 858
parsl.dataflow.dflow DEBUG: Task 858 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 5c2979fed58c7a3a558954e25c67fd5f
parsl.dataflow.memoization INFO: Task 859 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088096c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 742
parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9daa037b-f4c0-44cc-a778-c2132f792500_characterizeImage_946074_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9daa037b-f4c0-44cc-a778-c2132f792500_characterizeImage_946074_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e7d03dc1-c2ab-48fb-afbe-127efcff763a_isr_946074_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e7d03dc1-c2ab-48fb-afbe-127efcff763a_isr_946074_24.stderr
parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81cb9a44536e39874cbea80cd38e3034 with result from task 849
parsl.dataflow.dflow DEBUG: Task 849 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 850 has memoization hash ac70a6e67873121245294679eff3611e
parsl.dataflow.memoization INFO: Task 850 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 743
parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/8a5704d7-4c31-49a1-84d6-ea94e0c4f8ea_characterizeImage_946181_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/8a5704d7-4c31-49a1-84d6-ea94e0c4f8ea_characterizeImage_946181_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/37a20b03-03c3-440c-85c5-544902099dd4_isr_946181_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/37a20b03-03c3-440c-85c5-544902099dd4_isr_946181_26.stderr
parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 751152bec16e31fd1abaf5c090e56f95 with result from task 855
parsl.dataflow.dflow DEBUG: Task 855 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 6e1773192a982e201f141b802d8b8c93
parsl.dataflow.memoization INFO: Task 856 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088093a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 744
parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/feb6e222-72d7-4abe-8c57-548ebc4facbf_characterizeImage_946182_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/feb6e222-72d7-4abe-8c57-548ebc4facbf_characterizeImage_946182_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b16e8c5-4260-4521-9ca6-36dd4bdbd539_isr_946182_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/2b16e8c5-4260-4521-9ca6-36dd4bdbd539_isr_946182_26.stderr
parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 512ca059f5944559b061ded0fdcbe5d6 with result from task 852
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:52 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow DEBUG: Task 852 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 3b0a726c94bdc48c993bff4ec59f021e
parsl.dataflow.memoization INFO: Task 853 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 745
parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d4533c05-6ac2-44d8-815c-acc89a6f318d_characterizeImage_946181_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d4533c05-6ac2-44d8-815c-acc89a6f318d_characterizeImage_946181_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/8e9f7def-0d65-4328-8b41-877a8394ace8_isr_946181_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/8e9f7def-0d65-4328-8b41-877a8394ace8_isr_946181_42.stderr
parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f1dca7955d16c11c9f4d51b2c170b531 with result from task 864
parsl.dataflow.dflow DEBUG: Task 864 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 865 has memoization hash f596b4f059e9c66aed10b695c05fdbed
parsl.dataflow.memoization INFO: Task 865 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 746
parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7e4b4116-f597-47b9-b881-cb6b5e7b5a2d_characterizeImage_946181_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7e4b4116-f597-47b9-b881-cb6b5e7b5a2d_characterizeImage_946181_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e50c1b84-2749-4f97-a3df-971bccdf3964_isr_946181_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/e50c1b84-2749-4f97-a3df-971bccdf3964_isr_946181_12.stderr
parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4baf5b6f0e3784430f4e63b5675cdb3 with result from task 870
parsl.dataflow.dflow DEBUG: Task 870 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 871 has memoization hash a9644ab5c2e0b1e3293ea5abf29abfa8
parsl.dataflow.memoization INFO: Task 871 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 747
parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c9ecbca7-9abb-44ff-b88a-f79f3584af08_characterizeImage_946182_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c9ecbca7-9abb-44ff-b88a-f79f3584af08_characterizeImage_946182_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/141310e1-8e56-4bee-9127-355dcb6806ef_isr_946182_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/141310e1-8e56-4bee-9127-355dcb6806ef_isr_946182_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:52 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:53 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9f090ae29978f6b05ae4a09ecbd002c with result from task 744
parsl.dataflow.dflow DEBUG: Task 744 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 1f46f2cdb354d37b2ed5ea03d40c7cc1
parsl.dataflow.memoization INFO: Task 745 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 748
parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d121fbfe-8999-4ed7-a292-24fee80d56b6_characterizeImage_946180_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d121fbfe-8999-4ed7-a292-24fee80d56b6_characterizeImage_946180_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3733df0e-3ba4-41ac-b9b6-d8fd427ca02c_isr_946180_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/3733df0e-3ba4-41ac-b9b6-d8fd427ca02c_isr_946180_34.stderr
parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 519f425da493c92364c0eabb2baaddcd with result from task 756
parsl.dataflow.dflow DEBUG: Task 756 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 757 has memoization hash a7da2c593a791623fe97e20134ba3c1f
parsl.dataflow.memoization INFO: Task 757 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 749
parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1a867483-4012-4c58-b03f-ec849a399645_characterizeImage_946181_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1a867483-4012-4c58-b03f-ec849a399645_characterizeImage_946181_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/acf64e70-5917-4f94-abc9-9ddbeddcb0d9_isr_946181_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/acf64e70-5917-4f94-abc9-9ddbeddcb0d9_isr_946181_47.stderr
parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85ab9f34485fd6f3fe1cfa7fe610c145 with result from task 750
parsl.dataflow.dflow DEBUG: Task 750 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 0228db358f915e65541e55b7ef246e52
parsl.dataflow.memoization INFO: Task 751 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 750
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/0c8c49fe-93e0-4d9e-bee3-7cee27d0eff3_characterizeImage_946074_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/0c8c49fe-93e0-4d9e-bee3-7cee27d0eff3_characterizeImage_946074_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e2b58cb3-4588-4a97-bf51-1d7f2bc8d8ae_isr_946074_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/e2b58cb3-4588-4a97-bf51-1d7f2bc8d8ae_isr_946074_49.stderr
parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb4b9e6fe72e24102589ab602b0d127c with result from task 753
parsl.dataflow.dflow DEBUG: Task 753 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 754 has memoization hash e147f025288badb0163fcf6e56ce1c7a
parsl.dataflow.memoization INFO: Task 754 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 751
parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/260be758-8f5d-4838-9695-5eb9498d6852_characterizeImage_946181_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/260be758-8f5d-4838-9695-5eb9498d6852_characterizeImage_946181_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82f91d2f-263e-4f1b-8661-9d9f5a9104ac_isr_946181_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/82f91d2f-263e-4f1b-8661-9d9f5a9104ac_isr_946181_1.stderr
parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb58e6a41da4011841ad156dafd133a7 with result from task 738
parsl.dataflow.dflow DEBUG: Task 738 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 739 has memoization hash e98a59f28377a4282dc4649d87f7339a
parsl.dataflow.memoization INFO: Task 739 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 752
parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/28f5fb91-bfa1-4bfd-843b-e56727af878f_characterizeImage_946073_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/28f5fb91-bfa1-4bfd-843b-e56727af878f_characterizeImage_946073_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23152ea4-930c-4120-8cd4-8d68ad1b0c8d_isr_946073_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/23152ea4-930c-4120-8cd4-8d68ad1b0c8d_isr_946073_7.stderr
parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a38de3fd74a31936ccbc9f66b7d8963 with result from task 735
parsl.dataflow.dflow DEBUG: Task 735 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 7f564aa939c016f0008aed9090566796
parsl.dataflow.memoization INFO: Task 736 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 753
parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6d65eb9e-220e-4fc1-8a16-592deb32ed81_characterizeImage_946072_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6d65eb9e-220e-4fc1-8a16-592deb32ed81_characterizeImage_946072_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/82b3c135-a00a-4608-be57-1d33cffdad2b_isr_946072_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/82b3c135-a00a-4608-be57-1d33cffdad2b_isr_946072_30.stderr
parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ba7d608442d1218ec02f2369e5ec1d9 with result from task 747
parsl.dataflow.dflow DEBUG: Task 747 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 44f681f9a7d6c767c5c6c5c84706010f
parsl.dataflow.memoization INFO: Task 748 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 754
parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1555dc34-c4c4-4d0e-8236-af86be5b4c82_characterizeImage_946181_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1555dc34-c4c4-4d0e-8236-af86be5b4c82_characterizeImage_946181_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/febfb9df-8a17-4dfc-957f-5638ed5d4d7d_isr_946181_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/febfb9df-8a17-4dfc-957f-5638ed5d4d7d_isr_946181_20.stderr
parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc3451326df6ad122539a6be7e045fbd with result from task 741
parsl.dataflow.dflow DEBUG: Task 741 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 742 has memoization hash a8a94d7ffa5f17c60e6c709c24e201b3
parsl.dataflow.memoization INFO: Task 742 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 755
parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/435ca46f-0b19-4837-910a-a1c38e47dff1_characterizeImage_946074_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/435ca46f-0b19-4837-910a-a1c38e47dff1_characterizeImage_946074_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/bcab3b52-02cb-4bc8-aed0-1025445fdf27_isr_946074_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/bcab3b52-02cb-4bc8-aed0-1025445fdf27_isr_946074_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:53 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:54 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 071baf99a9a45ae57e840e8f69b4865c with result from task 777
parsl.dataflow.dflow DEBUG: Task 777 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 778 has memoization hash eb817ddd3e6d12ea8d60c15dfd596949
parsl.dataflow.memoization INFO: Task 778 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 756
parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/bfe2cf0d-0948-40a8-a6c2-0957ef50f3b8_characterizeImage_946072_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/bfe2cf0d-0948-40a8-a6c2-0957ef50f3b8_characterizeImage_946072_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2c60f8fa-1306-45eb-a26c-8484da33ce51_isr_946072_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/2c60f8fa-1306-45eb-a26c-8484da33ce51_isr_946072_58.stderr
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3524b97997fee1254c0b82d05af434c2 with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 8041b28016a8d2f464761a3b12979e09
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 757
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/639afbeb-6d5d-4457-ba27-a3cf495583a7_calibrate_946182_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/639afbeb-6d5d-4457-ba27-a3cf495583a7_calibrate_946182_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a0e09ed3-555d-4572-aa0b-01f9edee5b6c_characterizeImage_946182_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a0e09ed3-555d-4572-aa0b-01f9edee5b6c_characterizeImage_946182_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:54 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:55 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79b0495cda5a3805e2dae5b1bc02204e with result from task 208
parsl.dataflow.dflow DEBUG: Task 208 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 41f6e4d423188d3fc0be5c9d49712f48
parsl.dataflow.memoization INFO: Task 209 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 758
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/988b7b6c-9bfc-4afc-9c41-3af82784535b_calibrate_946182_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/988b7b6c-9bfc-4afc-9c41-3af82784535b_calibrate_946182_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/50a6ceab-81d4-43fd-b170-91bf84146ed8_characterizeImage_946182_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/50a6ceab-81d4-43fd-b170-91bf84146ed8_characterizeImage_946182_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:55 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8103fa90fafe3102942d199b33133fe9 with result from task 1050
parsl.dataflow.dflow DEBUG: Task 1050 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash a705677739dd7451d98c24a86a4a9807
parsl.dataflow.memoization INFO: Task 1051 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 759
parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ff6c9446-9310-4245-b279-aa4d20e3e449_characterizeImage_946181_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ff6c9446-9310-4245-b279-aa4d20e3e449_characterizeImage_946181_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b4b8f4f6-d9a0-4b7f-a5d6-e821f8564b70_isr_946181_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/b4b8f4f6-d9a0-4b7f-a5d6-e821f8564b70_isr_946181_4.stderr
parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6ec0e9b30f72fe919864f88495eb95 with result from task 1089
parsl.dataflow.dflow DEBUG: Task 1089 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 373dc6c6b9b68b99a452af0252ece118
parsl.dataflow.memoization INFO: Task 1090 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 760
parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/29221276-e60a-4d2e-a3a2-251e9e3d5458_characterizeImage_946181_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/29221276-e60a-4d2e-a3a2-251e9e3d5458_characterizeImage_946181_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5050735e-e323-4448-bf72-e317b301dd76_isr_946181_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/5050735e-e323-4448-bf72-e317b301dd76_isr_946181_8.stderr
parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d7e88f85791cf5f814d1fe8255335d1 with result from task 1047
parsl.dataflow.dflow DEBUG: Task 1047 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 81723455dca69cd3cb55d980634f1554
parsl.dataflow.memoization INFO: Task 1048 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 761
parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0cf01beb-7555-40a8-88ba-e574eae99326_characterizeImage_946181_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0cf01beb-7555-40a8-88ba-e574eae99326_characterizeImage_946181_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/def43128-3d27-4710-bde4-f58e064b6ed8_isr_946181_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/def43128-3d27-4710-bde4-f58e064b6ed8_isr_946181_11.stderr
parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54979c5d0aaaaf91a8164e65d241d50b with result from task 1056
parsl.dataflow.dflow DEBUG: Task 1056 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash f27922de4804fb1f3e62b95f21bb9369
parsl.dataflow.memoization INFO: Task 1057 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 762
parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/09e0ae8c-33c6-4d6a-ad9c-e5f3aaa044e5_characterizeImage_946182_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/09e0ae8c-33c6-4d6a-ad9c-e5f3aaa044e5_characterizeImage_946182_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/78b8a866-1f10-4b93-849f-d5b4782882e7_isr_946182_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/78b8a866-1f10-4b93-849f-d5b4782882e7_isr_946182_38.stderr
parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea212991513fd6a9f33a2b0065ae2b1 with result from task 1071
parsl.dataflow.dflow DEBUG: Task 1071 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 1100cb8383d86df548b5dddb28031caf
parsl.dataflow.memoization INFO: Task 1072 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 763
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7846b11a-c2f3-4d69-8f00-376bfb855509_characterizeImage_946180_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7846b11a-c2f3-4d69-8f00-376bfb855509_characterizeImage_946180_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b90b0567-cc32-4a9b-b6ad-852a4d28f2b6_isr_946180_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/b90b0567-cc32-4a9b-b6ad-852a4d28f2b6_isr_946180_35.stderr
parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 236d6889695f0c7da0060bf420478d49 with result from task 1062
parsl.dataflow.dflow DEBUG: Task 1062 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 428995436f374ae9cc84bc3ef56ebc0f
parsl.dataflow.memoization INFO: Task 1063 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 764
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d69402cc-b3df-4d95-a5db-13a44f5ceadc_isr_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/d69402cc-b3df-4d95-a5db-13a44f5ceadc_isr_946073_2.stderr
parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3369fe81d6e7829b16ca54e643bbe3b3 with result from task 1059
parsl.dataflow.dflow DEBUG: Task 1059 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 5e798cf8056ef99def8787d62ef757ec
parsl.dataflow.memoization INFO: Task 1060 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 765
parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7295d796-0203-44c6-8275-d20462ef5726_characterizeImage_946181_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7295d796-0203-44c6-8275-d20462ef5726_characterizeImage_946181_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7f7368aa-4b91-465b-a72b-8afca85924a7_isr_946181_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/7f7368aa-4b91-465b-a72b-8afca85924a7_isr_946181_17.stderr
parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7b746b512bd5a49122fda5508ffb1e6 with result from task 1083
parsl.dataflow.dflow DEBUG: Task 1083 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 749f8639497d68a486157b1a31639f4b
parsl.dataflow.memoization INFO: Task 1084 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 766
parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cbe95b10-0732-4b92-908b-2ea4b7d03bbd_characterizeImage_946072_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cbe95b10-0732-4b92-908b-2ea4b7d03bbd_characterizeImage_946072_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3ce09ee6-7373-4a6d-8aa4-cabb54bee44b_isr_946072_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946072/3ce09ee6-7373-4a6d-8aa4-cabb54bee44b_isr_946072_6.stderr
parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ebdcfa697c302b83a3b3b76905da0b4 with result from task 1080
parsl.dataflow.dflow DEBUG: Task 1080 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash 600a7870e18e7b1616a82bb00622fc19
parsl.dataflow.memoization INFO: Task 1081 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 767
parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6580c083-7cde-4e89-ae2c-052fc200a461_characterizeImage_946182_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6580c083-7cde-4e89-ae2c-052fc200a461_characterizeImage_946182_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0379898b-d14b-422d-ae43-8f6777be9e58_isr_946182_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/0379898b-d14b-422d-ae43-8f6777be9e58_isr_946182_14.stderr
parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87f987aac4c18d3f35c39703879b4525 with result from task 1065
parsl.dataflow.dflow DEBUG: Task 1065 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash dd0e2a4c51bab36eddff665f2ab68076
parsl.dataflow.memoization INFO: Task 1066 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 768
parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3da21804-299e-416c-926e-4e07eab48d20_characterizeImage_946180_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3da21804-299e-416c-926e-4e07eab48d20_characterizeImage_946180_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d6d9786f-78f2-43c0-9b34-6789ba7e4985_isr_946180_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/d6d9786f-78f2-43c0-9b34-6789ba7e4985_isr_946180_57.stderr
parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97111f522a5feb7d18e92b28d4758bb5 with result from task 1053
parsl.dataflow.dflow DEBUG: Task 1053 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 7830e813e9715bd53440b56017cd6fc7
parsl.dataflow.memoization INFO: Task 1054 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 769
parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/49c7891e-fed8-4199-9a0e-cc42c76f81e4_characterizeImage_946182_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/49c7891e-fed8-4199-9a0e-cc42c76f81e4_characterizeImage_946182_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb3943d6-6b36-4fb3-ac18-00303f67c1c8_isr_946182_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946182/eb3943d6-6b36-4fb3-ac18-00303f67c1c8_isr_946182_41.stderr
parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c2a3612fe586b7ca8225ca1546a8d61 with result from task 1077
parsl.dataflow.dflow DEBUG: Task 1077 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash 01a82f898d24c4ae82f93def370e558a
parsl.dataflow.memoization INFO: Task 1078 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ebba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 770
parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e53092e6-be8c-47b0-8671-babe6e728f16_characterizeImage_946074_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e53092e6-be8c-47b0-8671-babe6e728f16_characterizeImage_946074_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fba59dfd-aaa5-4526-be5a-0e7874c91e52_isr_946074_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/fba59dfd-aaa5-4526-be5a-0e7874c91e52_isr_946074_8.stderr
parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5468312458ffcff43289cd9a923956f9 with result from task 1044
parsl.dataflow.dflow DEBUG: Task 1044 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash d0009f5052bed072ef4e59cb7e969eb6
parsl.dataflow.memoization INFO: Task 1045 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 771
parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/baa87add-d414-4e53-a350-38226fb50489_characterizeImage_946181_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/baa87add-d414-4e53-a350-38226fb50489_characterizeImage_946181_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/71d6bfa2-00e7-428b-b3d3-fc1ed2996b1b_isr_946181_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/71d6bfa2-00e7-428b-b3d3-fc1ed2996b1b_isr_946181_40.stderr
parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4f9b80dbed443435a70ad477f4f1546 with result from task 1086
parsl.dataflow.dflow DEBUG: Task 1086 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash b1ace59dc4d991644bbc7731f993ba1d
parsl.dataflow.memoization INFO: Task 1087 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 772
parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ee13c141-2b31-450a-949b-5a4898c48c27_characterizeImage_946073_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ee13c141-2b31-450a-949b-5a4898c48c27_characterizeImage_946073_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/aaad56f7-bc85-4e2d-b21e-b1beb3e0ae34_isr_946073_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946073/aaad56f7-bc85-4e2d-b21e-b1beb3e0ae34_isr_946073_12.stderr
parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81b54ef4dbba6963513287e25922aa02 with result from task 1068
parsl.dataflow.dflow DEBUG: Task 1068 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 908b134de311e57afc78a32f8e221144
parsl.dataflow.memoization INFO: Task 1069 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 773
parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/78166bed-694c-4733-b7c1-19e01085530a_characterizeImage_946074_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/78166bed-694c-4733-b7c1-19e01085530a_characterizeImage_946074_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b054a092-be49-4975-8c11-49c03f8053e8_isr_946074_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/b054a092-be49-4975-8c11-49c03f8053e8_isr_946074_62.stderr
parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 497e3ef5b8735e282ccb4b80321289a2 with result from task 1074
parsl.dataflow.dflow DEBUG: Task 1074 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash 5025bd1b8880820082d3b57d0b569dd8
parsl.dataflow.memoization INFO: Task 1075 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 774
parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e680f82d-4f07-4c8d-aa88-e649c7b35dda_characterizeImage_946180_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e680f82d-4f07-4c8d-aa88-e649c7b35dda_characterizeImage_946180_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce2d451d-4e54-4888-9d43-f8e3e5653b4d_isr_946180_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946180/ce2d451d-4e54-4888-9d43-f8e3e5653b4d_isr_946180_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:56 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df7062295a50ff85c6672868beac708a with result from task 367
parsl.dataflow.dflow DEBUG: Task 367 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 14c6c3072a559e399bc8bdd6f227c3f5
parsl.dataflow.memoization INFO: Task 368 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 775
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/3d52e8ef-a22c-4215-9d53-2ca44a0aaf95_calibrate_946074_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/3d52e8ef-a22c-4215-9d53-2ca44a0aaf95_calibrate_946074_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/65e9533b-d25a-48da-95f6-f5816579e56b_characterizeImage_946074_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/65e9533b-d25a-48da-95f6-f5816579e56b_characterizeImage_946074_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:56 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:57 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:57 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f44eeb567c9248f95119affc71f159a6 with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash a601fe057b2501ecbc7fe7041274f3d8
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 776
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/83cc5624-3706-4875-9c48-3b9c275ba9f8_calibrate_946182_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/83cc5624-3706-4875-9c48-3b9c275ba9f8_calibrate_946182_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7fc76e6f-d585-4d9e-b3ed-f32eee3ecebc_characterizeImage_946182_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7fc76e6f-d585-4d9e-b3ed-f32eee3ecebc_characterizeImage_946182_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:58 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55260c8ff7606420a121d525265b65a6 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 7b8f24f644278b4b83c56b7edf8e4ef4
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 777
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d28d81ba-50d8-4e80-9878-1528e8c9f724_calibrate_946180_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d28d81ba-50d8-4e80-9878-1528e8c9f724_calibrate_946180_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/063a5d36-3355-4d30-b09c-221e4f424059_characterizeImage_946180_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/063a5d36-3355-4d30-b09c-221e4f424059_characterizeImage_946180_48.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 610 try 1 failed
parsl.dataflow.dflow ERROR: Task 610 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c72e7ce086c29ce87447fb99659d9d88 with result from task 610
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 611 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 611 try 0 failed
parsl.dataflow.dflow INFO: Task 611 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/0d7e6a0b-4f1d-49d9-8b9d-1b7f39360279_calibrate_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/0d7e6a0b-4f1d-49d9-8b9d-1b7f39360279_calibrate_946072_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a560eb6c-d3a3-4b7a-b7d5-b9d01ce078ce_characterizeImage_946072_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 409 try 1 failed
parsl.dataflow.dflow ERROR: Task 409 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 42d77eb14a3a84b3544107491ef2a6fb with result from task 409
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 410 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 410 try 0 failed
parsl.dataflow.dflow INFO: Task 410 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/4e6b14a0-db18-4899-beaa-6496d935fe2c_calibrate_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/4e6b14a0-db18-4899-beaa-6496d935fe2c_calibrate_946182_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1721a92a-8e5f-4286-ab97-09a257402ecd_characterizeImage_946182_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:58 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:59 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e72835816af6ed883c10843d976ac6a1 with result from task 376
parsl.dataflow.dflow DEBUG: Task 376 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 3d2c8ed1c7bbe8111c6338af05b5b21d
parsl.dataflow.memoization INFO: Task 377 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 778
parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/4e025109-feef-4cf7-a2af-a40a5b17d552_calibrate_946180_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/4e025109-feef-4cf7-a2af-a40a5b17d552_calibrate_946180_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/56e00260-8f2f-4e62-841d-091da0f628ce_characterizeImage_946180_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/56e00260-8f2f-4e62-841d-091da0f628ce_characterizeImage_946180_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:59 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:36:59 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 634 try 1 failed
parsl.dataflow.dflow ERROR: Task 634 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a99afa4eb9ed6ff4f0daaa813b24a03 with result from task 634
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 635 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 635 try 0 failed
parsl.dataflow.dflow INFO: Task 635 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/32fd8352-ce2d-4c5e-b5eb-8be3a3f0744e_calibrate_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/32fd8352-ce2d-4c5e-b5eb-8be3a3f0744e_calibrate_946074_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d884a091-f2f3-4374-8e54-c53758bafbf3_characterizeImage_946074_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 368
parsl.dataflow.strategy DEBUG: Executor multi has 368 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:37:00 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 425d65f3c494c3179eb48bc25bea3135 with result from task 229
parsl.dataflow.dflow DEBUG: Task 229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 6cde93f3bfc937e2544de28d03d79a91
parsl.dataflow.memoization INFO: Task 230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 779
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ba5007ab-3d3e-490c-95cd-97c6c4075039_calibrate_946181_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ba5007ab-3d3e-490c-95cd-97c6c4075039_calibrate_946181_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/10b39b23-a282-48a8-9f52-e9eb9454c857_characterizeImage_946181_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/10b39b23-a282-48a8-9f52-e9eb9454c857_characterizeImage_946181_32.stderr
2024-03-13 21:37:00 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:00 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 739575a09e876504e2c87cf7538ae332 with result from task 355
parsl.dataflow.dflow DEBUG: Task 355 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 356 has memoization hash cad697e6f25539291fad1f46fc7b91fb
parsl.dataflow.memoization INFO: Task 356 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 780
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5afac2b1-0e03-4066-b5c2-a28862b810f8_calibrate_946074_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5afac2b1-0e03-4066-b5c2-a28862b810f8_calibrate_946074_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fab348cd-12ca-4d18-957f-f4a852fa286e_characterizeImage_946074_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fab348cd-12ca-4d18-957f-f4a852fa286e_characterizeImage_946074_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:01 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4aca5057043992708ab01972a0a4bb25 with result from task 412
parsl.dataflow.dflow DEBUG: Task 412 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 730609c62dc7b379ecdfb8b06ca747c2
parsl.dataflow.memoization INFO: Task 413 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 781
parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2ac1d876-262c-4775-9f6d-d137e90308aa_calibrate_946072_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2ac1d876-262c-4775-9f6d-d137e90308aa_calibrate_946072_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/492a7a9b-9283-4cbb-9e16-e071bc0ab497_characterizeImage_946072_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/492a7a9b-9283-4cbb-9e16-e071bc0ab497_characterizeImage_946072_47.stderr
parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97670852ae2c8fbcd5d5ad6406a9c349 with result from task 235
parsl.dataflow.dflow DEBUG: Task 235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 311e8db663f8c37dec9f058f373655b7
parsl.dataflow.memoization INFO: Task 236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 782
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d2b34aa1-4bc7-4c7a-8420-17c00b674e0a_calibrate_946072_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d2b34aa1-4bc7-4c7a-8420-17c00b674e0a_calibrate_946072_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/edfbf97e-c3c1-451b-8ab3-b79694d159f9_characterizeImage_946072_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/edfbf97e-c3c1-451b-8ab3-b79694d159f9_characterizeImage_946072_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:01 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:02 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19e9c259b7df45f83caa30592ba44ed8 with result from task 154
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:02 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash d21d91b14e77e07f24dac16a1329fe82
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 783
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3ee67fcc-0c1a-4459-8d34-53c9c290eef5_calibrate_946072_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3ee67fcc-0c1a-4459-8d34-53c9c290eef5_calibrate_946072_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/751704a5-db7f-4f24-90e9-62eefa745427_characterizeImage_946072_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/751704a5-db7f-4f24-90e9-62eefa745427_characterizeImage_946072_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:02 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:03 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 612cba8d44b5f75241669d638631c5ca with result from task 274
parsl.dataflow.dflow DEBUG: Task 274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 873e2b221879b656d468b1809d7e2429
parsl.dataflow.memoization INFO: Task 275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 784
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1f380f6a-43af-4e59-9d1d-f7661bdfab50_calibrate_946073_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1f380f6a-43af-4e59-9d1d-f7661bdfab50_calibrate_946073_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/803bc082-9adb-4b3b-9432-e916231308bb_characterizeImage_946073_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/803bc082-9adb-4b3b-9432-e916231308bb_characterizeImage_946073_1.stderr
parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d9f419793b2a892f487ffedabca914d with result from task 712
parsl.dataflow.dflow DEBUG: Task 712 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 713 has memoization hash bd1352a0f246414380e61155a10ebb1d
parsl.dataflow.memoization INFO: Task 713 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 785
parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8ab0cfb4-a975-435e-a63d-df8cbd289db9_calibrate_946074_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8ab0cfb4-a975-435e-a63d-df8cbd289db9_calibrate_946074_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/6fd98e3f-49ca-4768-8a44-6861de632b21_characterizeImage_946074_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/6fd98e3f-49ca-4768-8a44-6861de632b21_characterizeImage_946074_21.stderr
parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 090b3341c3c13a62b40b2772fc379550 with result from task 361
parsl.dataflow.dflow DEBUG: Task 361 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 9948b53a8e961d484612b406b83db30c
parsl.dataflow.memoization INFO: Task 362 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 786
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b66cc311-6663-4bd2-a18b-4375947d7630_calibrate_946180_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b66cc311-6663-4bd2-a18b-4375947d7630_calibrate_946180_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b6d4662e-3c43-44c9-a36f-edaaadaee3f7_characterizeImage_946180_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b6d4662e-3c43-44c9-a36f-edaaadaee3f7_characterizeImage_946180_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:03 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:04 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6de56ef2acd9bfc36c8f7da7e9d4366 with result from task 1099
parsl.dataflow.dflow DEBUG: Task 1099 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash dc49950cc97ee6504203d566e5bb45ca
parsl.dataflow.memoization INFO: Task 1100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 787
parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/586d19fe-ad8e-4358-8176-030930a606e0_calibrate_946073_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/586d19fe-ad8e-4358-8176-030930a606e0_calibrate_946073_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/64b53b62-ca49-4924-89b3-3736b180beca_characterizeImage_946073_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/64b53b62-ca49-4924-89b3-3736b180beca_characterizeImage_946073_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:04 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:04 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:05 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b823a5839db002a125ae7f31b7729dc4 with result from task 1110
parsl.dataflow.dflow DEBUG: Task 1110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash ec8abeea568b4f48f51c925f2e66fc9f
parsl.dataflow.memoization INFO: Task 1111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 788
parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/299a3ff9-a3f8-403e-95b6-d3db96ff404f_characterizeImage_946074_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/299a3ff9-a3f8-403e-95b6-d3db96ff404f_characterizeImage_946074_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/80016d4d-d8ad-4b90-83c0-85551206b34c_isr_946074_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/80016d4d-d8ad-4b90-83c0-85551206b34c_isr_946074_36.stderr
parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc86d60efe8b22d6b39a06b378ccc00f with result from task 1107
parsl.dataflow.dflow DEBUG: Task 1107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash dcbdca5a1b63e188215f50237dfed6e8
parsl.dataflow.memoization INFO: Task 1108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 789
parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9e21cdcc-0b08-449b-bc21-3678f759d483_characterizeImage_946074_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9e21cdcc-0b08-449b-bc21-3678f759d483_characterizeImage_946074_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c183a6f7-f691-4811-819b-83e332d19f34_isr_946074_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946074/c183a6f7-f691-4811-819b-83e332d19f34_isr_946074_55.stderr
parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3fec2f5caeddf272c67ad5ae20f72be6 with result from task 1113
parsl.dataflow.dflow DEBUG: Task 1113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 6d4b7c05233b7692d820c889372b7da7
parsl.dataflow.memoization INFO: Task 1114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308736160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 790
parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/357abdff-b8be-4bfa-a2be-dca6fd5e322e_characterizeImage_946181_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/357abdff-b8be-4bfa-a2be-dca6fd5e322e_characterizeImage_946181_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9ae62eca-7fe4-4a20-b9bb-605573883ed3_isr_946181_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/isr/946181/9ae62eca-7fe4-4a20-b9bb-605573883ed3_isr_946181_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:05 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1063 try 0 failed
parsl.dataflow.dflow INFO: Task 1063 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 428995436f374ae9cc84bc3ef56ebc0f
parsl.dataflow.memoization INFO: Task 1063 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1063 try 1 launched on executor multi with executor id 791
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stderr
parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c44778d4174e164eb24adf419efd6bf with result from task 383
parsl.dataflow.dflow DEBUG: Task 383 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a02c7b00-1b74-4796-918c-b0a1183a7434_calibrate_946073_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a02c7b00-1b74-4796-918c-b0a1183a7434_calibrate_946073_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:06 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2368c6855749594972a03fdc4c8268dc with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash bfeb060a50338ae21ca86561743bae94
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 792
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c7e540ff-5e85-49ce-b71a-1c3519b32837_calibrate_946073_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c7e540ff-5e85-49ce-b71a-1c3519b32837_calibrate_946073_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7879cc40-cc26-416e-8d07-9cfdc544e561_characterizeImage_946073_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7879cc40-cc26-416e-8d07-9cfdc544e561_characterizeImage_946073_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:06 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:06 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b007e7c42c65b54e5f18ec773d20da03 with result from task 445
parsl.dataflow.dflow DEBUG: Task 445 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 446 has memoization hash b1a16d838ed4d4c9f16ff76260413b51
parsl.dataflow.memoization INFO: Task 446 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 793
parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3e3d2c6a-87e8-46c4-a8ff-1f3ec20e991d_calibrate_946180_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3e3d2c6a-87e8-46c4-a8ff-1f3ec20e991d_calibrate_946180_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/766ef5b8-ed08-405a-a05f-06db471be75c_characterizeImage_946180_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/766ef5b8-ed08-405a-a05f-06db471be75c_characterizeImage_946180_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:07 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:07 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67acecee43ae2f502be8f18a65d58b36 with result from task 188
parsl.dataflow.dflow DEBUG: Task 188 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/55a2c953-0556-4c32-80b8-25e250833647_calibrate_946073_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/55a2c953-0556-4c32-80b8-25e250833647_calibrate_946073_21.stderr
parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2daa3553a5930b5ef916193a0cecd964 with result from task 220
parsl.dataflow.dflow DEBUG: Task 220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 073d26dd59677e10c28cdded741ed71a
parsl.dataflow.memoization INFO: Task 221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 794
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e4907077-9308-4e2f-a362-c1c89320c567_calibrate_946181_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e4907077-9308-4e2f-a362-c1c89320c567_calibrate_946181_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c941b2a8-046d-49b4-9f82-89df9237a447_characterizeImage_946181_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c941b2a8-046d-49b4-9f82-89df9237a447_characterizeImage_946181_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:08 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e64aa16172916a401160ddcb62cdd6dd with result from task 1039
parsl.dataflow.dflow DEBUG: Task 1039 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash c1bacd1f5cae6224ff808bb4f8fc08ec
parsl.dataflow.memoization INFO: Task 1040 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 795
parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/fec42c69-dea4-47e4-82a6-5a2624cde54e_calibrate_946181_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/fec42c69-dea4-47e4-82a6-5a2624cde54e_calibrate_946181_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd678ea9-bb1a-4db3-8f71-30f75c92c654_characterizeImage_946181_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd678ea9-bb1a-4db3-8f71-30f75c92c654_characterizeImage_946181_16.stderr
parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 063fbc5b8951219ee24041efdb16b3e7 with result from task 847
parsl.dataflow.dflow DEBUG: Task 847 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 2c0f5289d3f29afc94118bd775730fc7
parsl.dataflow.memoization INFO: Task 848 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 796
parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/20e219e4-5eeb-41df-93c9-5b1105f9438d_calibrate_946072_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/20e219e4-5eeb-41df-93c9-5b1105f9438d_calibrate_946072_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cd06ad3a-505f-4fc2-8b0e-ef53749be83b_characterizeImage_946072_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cd06ad3a-505f-4fc2-8b0e-ef53749be83b_characterizeImage_946072_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:08 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3d80aaaf0287b65af5e1a1bccdb15b with result from task 349
parsl.dataflow.dflow DEBUG: Task 349 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 350 has memoization hash e75e729f31232077740346993b78e279
parsl.dataflow.memoization INFO: Task 350 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 797
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/54a733a9-179a-4192-8e47-ce4ecee2c084_calibrate_946180_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/54a733a9-179a-4192-8e47-ce4ecee2c084_calibrate_946180_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/67493d7b-ed84-4557-842a-8c6fe0b6ee32_characterizeImage_946180_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/67493d7b-ed84-4557-842a-8c6fe0b6ee32_characterizeImage_946180_56.stderr
parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6674d81e806488cc005974102b2a5d8b with result from task 715
parsl.dataflow.dflow DEBUG: Task 715 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 2d8879e9591810c123f4f6aa64a77429
parsl.dataflow.memoization INFO: Task 716 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308937f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 798
parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/5bca9aa6-bbf7-4b51-9935-6e9cab1c2abe_calibrate_946073_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/5bca9aa6-bbf7-4b51-9935-6e9cab1c2abe_calibrate_946073_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/420dccb1-5e65-41e3-a764-64e8528e1378_characterizeImage_946073_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/420dccb1-5e65-41e3-a764-64e8528e1378_characterizeImage_946073_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:08 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23825408a6acbc5cf5271b91e255587e with result from task 217
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:09 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow DEBUG: Task 217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 5f27581a6e3c98e812c078bd65509753
parsl.dataflow.memoization INFO: Task 218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 799
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/36b96c52-c7d9-4119-a273-d33bef83809d_calibrate_946180_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/36b96c52-c7d9-4119-a273-d33bef83809d_calibrate_946180_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/f709fd05-f342-4806-b594-aa9a967f3afa_characterizeImage_946180_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/f709fd05-f342-4806-b594-aa9a967f3afa_characterizeImage_946180_39.stderr
parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7a7bc282c91a52779e4960956b9dcf9 with result from task 232
parsl.dataflow.dflow DEBUG: Task 232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 020961893ee8a2bd61f7a3be1ac98236
parsl.dataflow.memoization INFO: Task 233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 800
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/73581bf2-4f2e-467b-b9b3-b2ab20ee4ffd_calibrate_946182_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/73581bf2-4f2e-467b-b9b3-b2ab20ee4ffd_calibrate_946182_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/849d1649-1873-4b73-8ab5-de3f6bf39320_characterizeImage_946182_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/849d1649-1873-4b73-8ab5-de3f6bf39320_characterizeImage_946182_22.stderr
parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3f31ec69f1619082a4efb6fc0124e06 with result from task 397
parsl.dataflow.dflow DEBUG: Task 397 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 398 has memoization hash b1201055449e40df6ce3dba31568be22
parsl.dataflow.memoization INFO: Task 398 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 801
parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/a4370244-97c4-4bfe-bf86-d3b9ff654cdb_calibrate_946181_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/a4370244-97c4-4bfe-bf86-d3b9ff654cdb_calibrate_946181_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/4f4655bd-21c5-4580-88ff-55bb602072d7_characterizeImage_946181_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/4f4655bd-21c5-4580-88ff-55bb602072d7_characterizeImage_946181_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:09 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11bbe06260aeba52be42154fa6f8e96a with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 8318b522fd6685d391fb399ed5a4ec21
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 802
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b4d30b18-0112-4e7e-9920-126421c5079a_calibrate_946073_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b4d30b18-0112-4e7e-9920-126421c5079a_calibrate_946073_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/70e3c3b5-5695-4850-a7ea-f4a00e122258_characterizeImage_946073_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/70e3c3b5-5695-4850-a7ea-f4a00e122258_characterizeImage_946073_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:10 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:10 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:11 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d04e4c23ad0f8f57312b392aff1eb89f with result from task 697
parsl.dataflow.dflow DEBUG: Task 697 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 698 has memoization hash e6036a015d6c495ddb3a22961fb73f5a
parsl.dataflow.memoization INFO: Task 698 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 803
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/52609732-02fe-41d5-b6b4-8772c9654b51_calibrate_946182_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/52609732-02fe-41d5-b6b4-8772c9654b51_calibrate_946182_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fddf67d9-c763-40c7-a76e-ade24e71f0da_characterizeImage_946182_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fddf67d9-c763-40c7-a76e-ade24e71f0da_characterizeImage_946182_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:11 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:11 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cd45a7ca4a9692db68dae392719568d with result from task 199
parsl.dataflow.dflow DEBUG: Task 199 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 82b69e9ced921447c47037903f7aa73f
parsl.dataflow.memoization INFO: Task 200 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 804
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9fc5c4b2-5048-440c-b8b7-fc438c34cbb9_calibrate_946181_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9fc5c4b2-5048-440c-b8b7-fc438c34cbb9_calibrate_946181_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c4dc3a13-9606-4b5d-ac1d-c904da17d158_characterizeImage_946181_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/c4dc3a13-9606-4b5d-ac1d-c904da17d158_characterizeImage_946181_5.stderr
parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff943726815c34d9ce1d912ecc6d51ce with result from task 322
parsl.dataflow.dflow DEBUG: Task 322 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 323 has memoization hash c809592a08e86bd3ba89f8b0cc3d2196
parsl.dataflow.memoization INFO: Task 323 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 805
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fafca06d-e7ef-490c-8135-2b1113ab22e2_calibrate_946180_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fafca06d-e7ef-490c-8135-2b1113ab22e2_calibrate_946180_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/13ca9d75-e554-4348-a5a0-774a1c4fc205_characterizeImage_946180_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/13ca9d75-e554-4348-a5a0-774a1c4fc205_characterizeImage_946180_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:12 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:12 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1a7c0d10550cfb533940261c25b04ae with result from task 433
parsl.dataflow.dflow DEBUG: Task 433 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 8a62266926da326509618938edaf1984
parsl.dataflow.memoization INFO: Task 434 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 806
parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9a6a4fac-6200-4688-aac9-0bbdfc20e5de_calibrate_946073_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9a6a4fac-6200-4688-aac9-0bbdfc20e5de_calibrate_946073_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7d1b3-0433-4872-a5c5-6360481de193_characterizeImage_946073_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7d1b3-0433-4872-a5c5-6360481de193_characterizeImage_946073_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:13 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:13 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d44f95cd04c00c186953d6714287b9cd with result from task 262
parsl.dataflow.dflow DEBUG: Task 262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 263 has memoization hash b784659c5fea48f1a197f4cef1612a2e
parsl.dataflow.memoization INFO: Task 263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 807
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/634567f4-b651-4e1d-a926-91b797abd8fe_calibrate_946074_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/634567f4-b651-4e1d-a926-91b797abd8fe_calibrate_946074_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9996e73d-2cf8-4cce-92c9-81e867cb3a66_characterizeImage_946074_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9996e73d-2cf8-4cce-92c9-81e867cb3a66_characterizeImage_946074_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:13 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:14 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 175be84982ada72c21568a9378530271 with result from task 334
parsl.dataflow.dflow DEBUG: Task 334 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 240e9885389c3b954adea3a84ade579d
parsl.dataflow.memoization INFO: Task 335 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 808
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a3c749b2-ffa6-4c7c-960c-14d83e06c6ad_calibrate_946180_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a3c749b2-ffa6-4c7c-960c-14d83e06c6ad_calibrate_946180_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3e9d4392-4196-458c-a2ce-c636a565ccbe_characterizeImage_946180_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3e9d4392-4196-458c-a2ce-c636a565ccbe_characterizeImage_946180_31.stderr
parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5953621bc1e55dff44e9eac6ebda7db with result from task 364
parsl.dataflow.dflow DEBUG: Task 364 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 365 has memoization hash c12268d4f7c7c34978fc12dd34c70f33
parsl.dataflow.memoization INFO: Task 365 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 809
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c77c0ec4-e10e-404e-b4fb-2f99d9b7b7f6_calibrate_946180_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c77c0ec4-e10e-404e-b4fb-2f99d9b7b7f6_calibrate_946180_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3f4df01d-20fe-4f07-b1d7-1c5890055e0e_characterizeImage_946180_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3f4df01d-20fe-4f07-b1d7-1c5890055e0e_characterizeImage_946180_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:14 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d78da5c2a5aebe20405ebd279bf9a5ea with result from task 421
parsl.dataflow.dflow DEBUG: Task 421 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 924b0d4008c52cfbaf178dbb6b6beb33
parsl.dataflow.memoization INFO: Task 422 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 810
parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/9594e122-6c99-4591-a498-aead2fdfcf04_calibrate_946072_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/9594e122-6c99-4591-a498-aead2fdfcf04_calibrate_946072_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5215bcec-ad78-4df0-9714-a1580e24986d_characterizeImage_946072_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5215bcec-ad78-4df0-9714-a1580e24986d_characterizeImage_946072_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:14 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc3955a6752b8e46a003a2446dbe4d5b with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 0b29d31e751de45428f2b3a8cef7151f
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 811
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c3b5377d-b626-4019-90a5-01c8be216821_calibrate_946181_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c3b5377d-b626-4019-90a5-01c8be216821_calibrate_946181_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/542783a3-58d3-47a7-9e0e-451e68c12c30_characterizeImage_946181_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/542783a3-58d3-47a7-9e0e-451e68c12c30_characterizeImage_946181_14.stderr
parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5841c1617d072c38b78e2b1b8493f4c0 with result from task 877
parsl.dataflow.dflow DEBUG: Task 877 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 220daf19531417b53011c7778b8ee1f7
parsl.dataflow.memoization INFO: Task 878 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 812
parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64a87fd3-6b56-4695-8f21-cdc97877b64c_calibrate_946073_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64a87fd3-6b56-4695-8f21-cdc97877b64c_calibrate_946073_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/be02f030-4f7a-412f-98ce-3441185164d9_characterizeImage_946073_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/be02f030-4f7a-412f-98ce-3441185164d9_characterizeImage_946073_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:15 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b77741570436052dbe01bf0554ffd820 with result from task 814
parsl.dataflow.dflow DEBUG: Task 814 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 5b3d1a65f460f2af1c216709b1c83aea
parsl.dataflow.memoization INFO: Task 815 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 813
parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2f1cf6ea-94c1-47d6-a986-379cf21ea46e_calibrate_946074_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2f1cf6ea-94c1-47d6-a986-379cf21ea46e_calibrate_946074_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5768c43c-dbbf-4115-a4e3-f382449fe3c1_characterizeImage_946074_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5768c43c-dbbf-4115-a4e3-f382449fe3c1_characterizeImage_946074_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:15 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6d5b8322957fd916c76b69a5e95bd6b with result from task 442
parsl.dataflow.dflow DEBUG: Task 442 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 443 has memoization hash cadcd7409b293d6f41cd20ddcbbe9a87
parsl.dataflow.memoization INFO: Task 443 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a420c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 814
parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24829295-9a55-49ce-93a6-6a076a962705_calibrate_946181_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24829295-9a55-49ce-93a6-6a076a962705_calibrate_946181_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e6e08228-697a-4fab-98b5-3ad893c8fd36_characterizeImage_946181_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e6e08228-697a-4fab-98b5-3ad893c8fd36_characterizeImage_946181_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:15 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:16 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad505255f365608a207cf7fea2b8df5 with result from task 1096
parsl.dataflow.dflow DEBUG: Task 1096 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash 0cf6da3659635e5b352f9a0d69408a30
parsl.dataflow.memoization INFO: Task 1097 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 815
parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0538d2a6-a5d3-4486-98ae-1ae64177181a_calibrate_946182_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0538d2a6-a5d3-4486-98ae-1ae64177181a_calibrate_946182_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/84967d92-62b0-4699-82ef-9fb2909b1708_characterizeImage_946182_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/84967d92-62b0-4699-82ef-9fb2909b1708_characterizeImage_946182_27.stderr
parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c20033bbddfabbbd29234f0d6856fe9 with result from task 385
parsl.dataflow.dflow DEBUG: Task 385 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 3bc7e944c7aa7d1b2b7fe967719bb8fb
parsl.dataflow.memoization INFO: Task 386 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 816
parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/2188cf7d-2573-405a-a2d8-0b33f2830a63_calibrate_946073_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/2188cf7d-2573-405a-a2d8-0b33f2830a63_calibrate_946073_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/50be2ad6-dc8b-44c3-8758-0800e69de90e_characterizeImage_946073_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/50be2ad6-dc8b-44c3-8758-0800e69de90e_characterizeImage_946073_60.stderr
parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ce630ed93644f88074b70b4e73fcc66 with result from task 487
parsl.dataflow.dflow DEBUG: Task 487 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 83a265744ac3a7e1dfa14d3c3d56c0be
parsl.dataflow.memoization INFO: Task 488 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 817
parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/30a27325-f7e1-4094-b6de-31f2150a4246_calibrate_946074_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/30a27325-f7e1-4094-b6de-31f2150a4246_calibrate_946074_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/038350f3-683b-40ba-9e53-a83a8e26715a_characterizeImage_946074_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/038350f3-683b-40ba-9e53-a83a8e26715a_characterizeImage_946074_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:16 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:17 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:17 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:17 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1063 try 1 failed
parsl.dataflow.dflow ERROR: Task 1063 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 428995436f374ae9cc84bc3ef56ebc0f with result from task 1063
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1064 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1064 try 0 failed
parsl.dataflow.dflow INFO: Task 1064 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b8eb4555-b9f9-42a2-a0f7-4173ab152c97_calibrate_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b8eb4555-b9f9-42a2-a0f7-4173ab152c97_calibrate_946073_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/cc4e838f-bbed-471d-8be5-9dd468eb44c7_characterizeImage_946073_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:18 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:18 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:19 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:19 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:19 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:20 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68419a9226602c1cca1524dc92c70942 with result from task 212
parsl.dataflow.dflow DEBUG: Task 212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c6fc4235-33a2-4e52-b278-a2c8e1d04454_calibrate_946072_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c6fc4235-33a2-4e52-b278-a2c8e1d04454_calibrate_946072_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:20 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:21 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:21 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:21 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63b0f17c59cbd1c8629cbe92936f6248 with result from task 691
parsl.dataflow.dflow DEBUG: Task 691 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 7873059b6c8e4473a87b6b566e4820d7
parsl.dataflow.memoization INFO: Task 692 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 818
parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7ae2b82-e2f7-4269-9859-dd25959bea25_calibrate_946073_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7ae2b82-e2f7-4269-9859-dd25959bea25_calibrate_946073_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a87b8a5c-69be-4866-94e7-6620c27fb126_characterizeImage_946073_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a87b8a5c-69be-4866-94e7-6620c27fb126_characterizeImage_946073_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:22 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:22 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:22 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:22 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9664f1458ee8899d6167cbc01521272 with result from task 817
parsl.dataflow.dflow DEBUG: Task 817 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 818 has memoization hash d070af9c8d546a07d88bad62aea0081a
parsl.dataflow.memoization INFO: Task 818 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 819
parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/461368d5-440d-4179-a78d-d2b83563179c_calibrate_946073_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/461368d5-440d-4179-a78d-d2b83563179c_calibrate_946073_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a15d7b41-e6cc-4bc2-a5b8-778e2545eaef_characterizeImage_946073_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a15d7b41-e6cc-4bc2-a5b8-778e2545eaef_characterizeImage_946073_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:22 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e61db68948d3e95ff656fec8acce2c3 with result from task 223
parsl.dataflow.dflow DEBUG: Task 223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 7e8c49a9108729e2a22e28966b79980a
parsl.dataflow.memoization INFO: Task 224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 820
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ff6a4915-b856-498c-858c-13428c36890e_calibrate_946181_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ff6a4915-b856-498c-858c-13428c36890e_calibrate_946181_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3f87ce3e-0000-457e-bd24-f69329b5d1d5_characterizeImage_946181_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3f87ce3e-0000-457e-bd24-f69329b5d1d5_characterizeImage_946181_28.stderr
parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51a94bb611e4d90390ac7631cc979e97 with result from task 718
parsl.dataflow.dflow DEBUG: Task 718 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 719 has memoization hash ae2534b71b8c398b687324ca8e928ac6
parsl.dataflow.memoization INFO: Task 719 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 821
parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/41cba618-a9ac-4c98-8227-4154630d5b08_calibrate_946072_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/41cba618-a9ac-4c98-8227-4154630d5b08_calibrate_946072_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/091e1f30-74de-45f9-aea1-a2c1e37b1e9d_characterizeImage_946072_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/091e1f30-74de-45f9-aea1-a2c1e37b1e9d_characterizeImage_946072_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:23 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:23 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b86defd228a3069f348f404ab1201e08 with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 60439821f9fee119861b70255522ed8a
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 822
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/aa8edcc2-dbaa-46bc-9bb4-562ab6cdd7aa_calibrate_946181_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/aa8edcc2-dbaa-46bc-9bb4-562ab6cdd7aa_calibrate_946181_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b7ee437-7916-4e8b-ac62-b211a224750a_characterizeImage_946181_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b7ee437-7916-4e8b-ac62-b211a224750a_characterizeImage_946181_46.stderr
parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84254d650f673c56d826400b1fe214c3 with result from task 205
parsl.dataflow.dflow DEBUG: Task 205 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 88dcfaaab25c1aecd38aa0985125a364
parsl.dataflow.memoization INFO: Task 206 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 823
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ddbae87f-6fc7-4dc3-a25c-b5ca3ba1f5e2_calibrate_946180_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ddbae87f-6fc7-4dc3-a25c-b5ca3ba1f5e2_calibrate_946180_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e1b422ba-5dda-4aa4-839f-308535753f22_characterizeImage_946180_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e1b422ba-5dda-4aa4-839f-308535753f22_characterizeImage_946180_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:23 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:24 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0bd8697054fce0923aef190c1f385cc with result from task 214
parsl.dataflow.dflow DEBUG: Task 214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 942c22038c4ba007e8147e373a92f08a
parsl.dataflow.memoization INFO: Task 215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 824
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/06d427da-f5ff-4404-b671-e806bf084464_calibrate_946180_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/06d427da-f5ff-4404-b671-e806bf084464_calibrate_946180_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d5b3e8fb-21bd-4e8b-8870-d13325eaaf97_characterizeImage_946180_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d5b3e8fb-21bd-4e8b-8870-d13325eaaf97_characterizeImage_946180_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:24 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9ac69d7d2a8c6deca540eaa3a9a682 with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash ce8465c298831705eee6329a5bd1f650
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 825
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/284841a8-11dc-4b35-b99f-7560fc645db2_calibrate_946072_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/284841a8-11dc-4b35-b99f-7560fc645db2_calibrate_946072_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/38079b15-5366-4b15-815e-6e03ba53ee61_characterizeImage_946072_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/38079b15-5366-4b15-815e-6e03ba53ee61_characterizeImage_946072_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:25 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 83daf6301e76f1d95372dc8ac5f95565 with result from task 475
parsl.dataflow.dflow DEBUG: Task 475 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 476 has memoization hash d7b9c87deaae3b76c56372e27122716e
parsl.dataflow.memoization INFO: Task 476 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 826
parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/08469bbc-5403-4dca-a9df-9d179766c75c_calibrate_946074_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/08469bbc-5403-4dca-a9df-9d179766c75c_calibrate_946074_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ba8e5fd3-0a67-4e84-8523-736d7b030f4f_characterizeImage_946074_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ba8e5fd3-0a67-4e84-8523-736d7b030f4f_characterizeImage_946074_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:25 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:26 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:26 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e3ff768b0c4c098da97ea6f40685b5e8 with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 1ea9840f3aefbe88c8ad885a93997988
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 827
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/42a75377-42a2-4986-ad03-59e0db69d793_calibrate_946182_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/42a75377-42a2-4986-ad03-59e0db69d793_calibrate_946182_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/882089e9-028e-45b2-a812-1a3442e5bae8_characterizeImage_946182_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/882089e9-028e-45b2-a812-1a3442e5bae8_characterizeImage_946182_43.stderr
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec32a15c8dc28c73e4436cf27f4b5e4 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash ec35b846ff3c4ab7d63e8b45265fb875
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 828
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/09006943-70cd-4b0b-bfd2-e826cb75286b_calibrate_946180_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/09006943-70cd-4b0b-bfd2-e826cb75286b_calibrate_946180_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/48e3e8e6-2cff-4ad3-9e85-48a6edb9d72e_characterizeImage_946180_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/48e3e8e6-2cff-4ad3-9e85-48a6edb9d72e_characterizeImage_946180_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:27 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 439a075cf4458aec7daed523980793da with result from task 226
parsl.dataflow.dflow DEBUG: Task 226 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 34d9117833686c7eba63a0c0d6f3684b
parsl.dataflow.memoization INFO: Task 227 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 829
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c68a99fa-c607-4cca-b058-a64ee5002385_calibrate_946074_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c68a99fa-c607-4cca-b058-a64ee5002385_calibrate_946074_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/db394b94-2e64-4fac-868c-a30f5bd62eec_characterizeImage_946074_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/db394b94-2e64-4fac-868c-a30f5bd62eec_characterizeImage_946074_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:27 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 643211f7ad10ad1e7eefc5dfda02300d with result from task 706
parsl.dataflow.dflow DEBUG: Task 706 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 707 has memoization hash a28fdf103b1c2c4dac1b924550b9c582
parsl.dataflow.memoization INFO: Task 707 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 830
parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/f9f90c98-f857-4d02-99a2-8cba5ea3eb31_calibrate_946074_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/f9f90c98-f857-4d02-99a2-8cba5ea3eb31_calibrate_946074_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de322350-1597-4a53-b0be-055098845296_characterizeImage_946074_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de322350-1597-4a53-b0be-055098845296_characterizeImage_946074_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:28 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37db362567fda8c3f87c408c4f54bdd6 with result from task 380
parsl.dataflow.dflow DEBUG: Task 380 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/97c1f05f-154a-4767-bab4-0c69fe695ccd_calibrate_946073_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/97c1f05f-154a-4767-bab4-0c69fe695ccd_calibrate_946073_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:28 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2984f6c53b66b56ed5e709a118e73af3 with result from task 574
parsl.dataflow.dflow DEBUG: Task 574 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 127adc499dc7d445ed19cb275c6220e6
parsl.dataflow.memoization INFO: Task 575 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 831
parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b01c19e6-e960-4d3d-9ea5-ee81582d989c_calibrate_946072_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b01c19e6-e960-4d3d-9ea5-ee81582d989c_calibrate_946072_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/485948a4-136e-4ca6-85d7-33e900ce4ac0_characterizeImage_946072_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/485948a4-136e-4ca6-85d7-33e900ce4ac0_characterizeImage_946072_59.stderr
parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79e884194ee90bf37fdfd25c5ac53066 with result from task 580
parsl.dataflow.dflow DEBUG: Task 580 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 581 has memoization hash d8562c02e152022d8f6f2750d9da3634
parsl.dataflow.memoization INFO: Task 581 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 832
parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/797ee7b9-b9d5-4a0b-bcc0-2eac949a52d6_calibrate_946072_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/797ee7b9-b9d5-4a0b-bcc0-2eac949a52d6_calibrate_946072_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0707ecfc-3a94-45b8-8fbd-453c339c670f_characterizeImage_946072_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0707ecfc-3a94-45b8-8fbd-453c339c670f_characterizeImage_946072_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:28 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6749c2160246c5bc4bb1bca45e48d88c with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 4a12b5760dd60055d6798cdce1c848df
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 833
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee2d2edf-3553-46f7-8abd-b5ea871c7289_calibrate_946180_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee2d2edf-3553-46f7-8abd-b5ea871c7289_calibrate_946180_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6e4e577b-e260-455c-aef3-0da6291a6c41_characterizeImage_946180_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6e4e577b-e260-455c-aef3-0da6291a6c41_characterizeImage_946180_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:29 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:29 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:30 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:30 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66eec8c889dc6f9f97e78823c36dae08 with result from task 196
parsl.dataflow.dflow DEBUG: Task 196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 197 has memoization hash fb4d8b1e43871b4c8c041d3047e6f695
parsl.dataflow.memoization INFO: Task 197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 834
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/905474e2-e315-4ee3-b8ce-7d15f38f1637_calibrate_946073_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/905474e2-e315-4ee3-b8ce-7d15f38f1637_calibrate_946073_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f4175c90-75a0-49e4-a71d-48b54bbb2d12_characterizeImage_946073_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f4175c90-75a0-49e4-a71d-48b54bbb2d12_characterizeImage_946073_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:31 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d8a763c3d6abdb297635f46daac598f with result from task 244
parsl.dataflow.dflow DEBUG: Task 244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 4d63a29217ce0354ea826ed11d3e401d
parsl.dataflow.memoization INFO: Task 245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 835
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/246598a3-1b2c-4e09-a8d7-c27b26d87261_calibrate_946072_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/246598a3-1b2c-4e09-a8d7-c27b26d87261_calibrate_946072_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/71b3c84d-2bae-4b84-be72-aab9af3ed4dd_characterizeImage_946072_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/71b3c84d-2bae-4b84-be72-aab9af3ed4dd_characterizeImage_946072_14.stderr
parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c32611fdd5c7dc965eab47de194ebee with result from task 694
parsl.dataflow.dflow DEBUG: Task 694 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 1dc3dd72310ffd1ca6fe9d8b1472291c
parsl.dataflow.memoization INFO: Task 695 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 836
parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/aa19bef3-5e40-4626-a662-640cfd97b65b_calibrate_946072_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/aa19bef3-5e40-4626-a662-640cfd97b65b_calibrate_946072_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0b7f3802-88c5-4450-85b4-45a68ae7d823_characterizeImage_946072_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/0b7f3802-88c5-4450-85b4-45a68ae7d823_characterizeImage_946072_9.stderr
parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eac5a9b68036748a2c29e95dee8acfec with result from task 652
parsl.dataflow.dflow DEBUG: Task 652 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 653 has memoization hash e3d5940fbf8da867a2a9c2cb4a0a05af
parsl.dataflow.memoization INFO: Task 653 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088efc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 837
parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/638353bf-fbfa-40a6-a91a-5cb8d3aaf19f_calibrate_946074_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/638353bf-fbfa-40a6-a91a-5cb8d3aaf19f_calibrate_946074_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d8d53ad7-8a9b-4d97-bc6f-e7e4b51f75b9_characterizeImage_946074_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/d8d53ad7-8a9b-4d97-bc6f-e7e4b51f75b9_characterizeImage_946074_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:31 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d245511e0ba090bbdaee89a925cb040 with result from task 589
parsl.dataflow.dflow DEBUG: Task 589 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 2a5a60345d9adce287849f106fc47bce
parsl.dataflow.memoization INFO: Task 590 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 838
parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ff4d4748-bf41-47eb-8c64-f6b57497f039_calibrate_946072_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ff4d4748-bf41-47eb-8c64-f6b57497f039_calibrate_946072_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/ea52618e-a48d-4ef8-aab1-c76f8fcb5ff1_characterizeImage_946072_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/ea52618e-a48d-4ef8-aab1-c76f8fcb5ff1_characterizeImage_946072_39.stderr
parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c13b606d337b3acc77ba5df5a9f51af with result from task 439
parsl.dataflow.dflow DEBUG: Task 439 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 440 has memoization hash a7060b1de6fa54ef049141d02156b124
parsl.dataflow.memoization INFO: Task 440 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 839
parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c2844cc6-72b2-4d10-98da-998d5c06c16e_calibrate_946182_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c2844cc6-72b2-4d10-98da-998d5c06c16e_calibrate_946182_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cf659be7-e4c8-41bd-830f-8ee86cfded05_characterizeImage_946182_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cf659be7-e4c8-41bd-830f-8ee86cfded05_characterizeImage_946182_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:31 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:32 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1024 try 0 failed
parsl.dataflow.dflow INFO: Task 1024 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash d78583f1500fc98cf98036eff9d2a88d
parsl.dataflow.memoization INFO: Task 1024 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1024 try 1 launched on executor multi with executor id 840
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:32 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:33 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb9fb2838f3e8dc44c98e5a6edbcb8d0 with result from task 931
parsl.dataflow.dflow DEBUG: Task 931 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 674b4e469285717323bcf8585980ce3c
parsl.dataflow.memoization INFO: Task 932 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 841
parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/471e3a08-9547-47d9-ab73-caf47c73ab1c_calibrate_946073_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/471e3a08-9547-47d9-ab73-caf47c73ab1c_calibrate_946073_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/0095620b-6bb3-42df-9560-652f25095933_characterizeImage_946073_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/0095620b-6bb3-42df-9560-652f25095933_characterizeImage_946073_53.stderr
parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af6ffe4dfca59c882bc2852b500e7751 with result from task 373
parsl.dataflow.dflow DEBUG: Task 373 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 269baf549f10c80de71493eadc475bbe
parsl.dataflow.memoization INFO: Task 374 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 842
parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/d25d88ab-f9e5-4d31-a1c1-5734af17c324_calibrate_946182_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/d25d88ab-f9e5-4d31-a1c1-5734af17c324_calibrate_946182_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/126109d9-74e6-419f-8923-c1993153b241_characterizeImage_946182_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/126109d9-74e6-419f-8923-c1993153b241_characterizeImage_946182_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:33 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33c0894883018b1b6b00a802c495a808 with result from task 733
parsl.dataflow.dflow DEBUG: Task 733 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 80fd570218dfc6e23035ba131faded40
parsl.dataflow.memoization INFO: Task 734 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 843
parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/95b6a829-51af-4941-b6ad-7be0a86d37a8_calibrate_946180_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/95b6a829-51af-4941-b6ad-7be0a86d37a8_calibrate_946180_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c6be59d9-fb3d-49c5-812b-401d5450925f_characterizeImage_946180_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c6be59d9-fb3d-49c5-812b-401d5450925f_characterizeImage_946180_41.stderr
parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3469fadc4511aaa56acdd47a925b3930 with result from task 601
parsl.dataflow.dflow DEBUG: Task 601 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 1e42776d9a33b75751653474d1c46365
parsl.dataflow.memoization INFO: Task 602 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 844
parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3a70abaa-3eef-46c7-8078-63d486b13b0d_calibrate_946180_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3a70abaa-3eef-46c7-8078-63d486b13b0d_calibrate_946180_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a8e3b530-91e6-4533-bf5b-2aea2e767769_characterizeImage_946180_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a8e3b530-91e6-4533-bf5b-2aea2e767769_characterizeImage_946180_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:34 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c1040564f2fcfa9b111bb2bc79c5911 with result from task 586
parsl.dataflow.dflow DEBUG: Task 586 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 6854ab86d62f9a3c99085ca30e2e6e36
parsl.dataflow.memoization INFO: Task 587 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 845
parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aa56f8bf-630e-42b5-9c93-a27bb3a74ce7_calibrate_946073_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aa56f8bf-630e-42b5-9c93-a27bb3a74ce7_calibrate_946073_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f26755f9-85ef-46d4-9b82-21ce4c6cfc3b_characterizeImage_946073_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f26755f9-85ef-46d4-9b82-21ce4c6cfc3b_characterizeImage_946073_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:34 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:35 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eeed018bae1e716cd203f418e0c8fbcd with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 81b9a62c098365c0998cb3106110aec7
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 846
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6a0b1576-d3eb-4923-9727-4b8240e5905d_calibrate_946072_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6a0b1576-d3eb-4923-9727-4b8240e5905d_calibrate_946072_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6628ce72-89bd-477e-bc87-3d3889c16a30_characterizeImage_946072_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6628ce72-89bd-477e-bc87-3d3889c16a30_characterizeImage_946072_11.stderr
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7df78bc8dd831e37dce203e10ed4954f with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4bbc053c-b961-4531-96f2-19aa8d7932fb_calibrate_946073_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4bbc053c-b961-4531-96f2-19aa8d7932fb_calibrate_946073_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:35 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:35 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 45695d6886db1a334c42d8b202fbc1b2 with result from task 1093
parsl.dataflow.dflow DEBUG: Task 1093 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash d96d13698ed0d43ce8a8b0ec4130e5f3
parsl.dataflow.memoization INFO: Task 1094 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523087349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 847
parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ffef99e6-3993-4211-bb64-09f442801d7b_calibrate_946072_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ffef99e6-3993-4211-bb64-09f442801d7b_calibrate_946072_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e2dbab3c-a2a5-4efa-825b-16d7b58013fe_characterizeImage_946072_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e2dbab3c-a2a5-4efa-825b-16d7b58013fe_characterizeImage_946072_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:36 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62d1647a09c23b7a747b5e14952fbfee with result from task 481
parsl.dataflow.dflow DEBUG: Task 481 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 482 has memoization hash e43ecd7d3062560e0ac59a72d7d92991
parsl.dataflow.memoization INFO: Task 482 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 848
parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/aa530f39-bfc4-42e8-99db-a9e61ee2d4a5_calibrate_946180_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/aa530f39-bfc4-42e8-99db-a9e61ee2d4a5_calibrate_946180_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/dbfd4482-be8e-413c-a8c9-4490a42a06d7_characterizeImage_946180_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/dbfd4482-be8e-413c-a8c9-4490a42a06d7_characterizeImage_946180_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:36 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 362
parsl.dataflow.strategy DEBUG: Executor multi has 362 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:37:37 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:37:37 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3681e3b80e11ae04afcda1596f51d633 with result from task 760
parsl.dataflow.dflow DEBUG: Task 760 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 8e65a00e658ad64af95b9778c2265020
parsl.dataflow.memoization INFO: Task 761 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 849
parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/46bd657a-ae11-4d97-b4cc-664a38fdcd1e_calibrate_946182_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/46bd657a-ae11-4d97-b4cc-664a38fdcd1e_calibrate_946182_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c23519c8-f300-428a-9dab-16a773a4f790_characterizeImage_946182_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c23519c8-f300-428a-9dab-16a773a4f790_characterizeImage_946182_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:37 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 991 try 0 failed
parsl.dataflow.dflow INFO: Task 991 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 95f2cab2908ff757b269ba6c1271bf58
parsl.dataflow.memoization INFO: Task 991 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 991 try 1 launched on executor multi with executor id 850
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stderr
parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 512180684046d13be1f25a67489c2862 with result from task 928
parsl.dataflow.dflow DEBUG: Task 928 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 929 has memoization hash 8257ee9da52cbd619b293f0440dc2377
parsl.dataflow.memoization INFO: Task 929 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 851
parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/99c9a565-929e-45df-8245-c78aa0063fa5_calibrate_946074_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/99c9a565-929e-45df-8245-c78aa0063fa5_calibrate_946074_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5e2a5a70-a3b3-46dd-9205-4fd133d36bcc_characterizeImage_946074_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5e2a5a70-a3b3-46dd-9205-4fd133d36bcc_characterizeImage_946074_13.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d88122435be93180e125a38d6da7f2f with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 6fbbc5f6e6da7a71b72037ac9a24f269
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 852
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3cc4b7e-0c2f-4100-8775-6dd441dcc658_calibrate_946072_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3cc4b7e-0c2f-4100-8775-6dd441dcc658_calibrate_946072_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cf003b05-e86e-45c7-adfb-cad778db261f_characterizeImage_946072_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cf003b05-e86e-45c7-adfb-cad778db261f_characterizeImage_946072_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:38 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22d73add482284b32267e1ccc73f90d8 with result from task 466
parsl.dataflow.dflow DEBUG: Task 466 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 0f7d7a01a72c6d8ab72dcbc2bc3ed4c7
parsl.dataflow.memoization INFO: Task 467 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 853
parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2fd81625-9859-4abd-a094-5d67ab913f03_calibrate_946072_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2fd81625-9859-4abd-a094-5d67ab913f03_calibrate_946072_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/171dac64-dfdd-4fb9-a233-edfd4fc4aedb_characterizeImage_946072_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/171dac64-dfdd-4fb9-a233-edfd4fc4aedb_characterizeImage_946072_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:38 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:39 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:39 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fdda165a83c37b2bd007fe42cc6021a with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash ed5999b4257041680125ec3148460545
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 854
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5e66cea8-687f-4dcc-8ea7-ff7edde1178e_calibrate_946182_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5e66cea8-687f-4dcc-8ea7-ff7edde1178e_calibrate_946182_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/eed47b7c-4ca0-4f99-b7ff-1e836b6e2192_characterizeImage_946182_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/eed47b7c-4ca0-4f99-b7ff-1e836b6e2192_characterizeImage_946182_21.stderr
parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc982968638e22f813bc859177322aac with result from task 727
parsl.dataflow.dflow DEBUG: Task 727 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 4630bfecad5a20df73a1d5204f6562d5
parsl.dataflow.memoization INFO: Task 728 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 855
parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/423716c4-6e40-4454-b911-21cb7abc175c_calibrate_946180_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/423716c4-6e40-4454-b911-21cb7abc175c_calibrate_946180_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ce612a0a-68a1-4201-b58b-57cd14f21363_characterizeImage_946180_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ce612a0a-68a1-4201-b58b-57cd14f21363_characterizeImage_946180_36.stderr
parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 960ea0a656f948baacb279eb72dec693 with result from task 661
parsl.dataflow.dflow DEBUG: Task 661 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 1933d0fd6881b92cc80c1df4d1b76134
parsl.dataflow.memoization INFO: Task 662 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 856
parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6f45e800-2dd6-4141-a402-5d8826cc6a00_calibrate_946073_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6f45e800-2dd6-4141-a402-5d8826cc6a00_calibrate_946073_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c64f463b-d22a-4f4e-ba2e-f79b1056ffa0_characterizeImage_946073_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c64f463b-d22a-4f4e-ba2e-f79b1056ffa0_characterizeImage_946073_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:39 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aec1e842c386684f4b120c666de5b696 with result from task 400
parsl.dataflow.dflow DEBUG: Task 400 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 9211f70f3b19b9a8cf0a77a183556d85
parsl.dataflow.memoization INFO: Task 401 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 857
parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/79aaef45-2b2e-45d9-85da-393ac89f2a0f_calibrate_946181_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/79aaef45-2b2e-45d9-85da-393ac89f2a0f_calibrate_946181_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7b22a60-f162-4ff3-8e30-292a3e977dd5_characterizeImage_946181_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7b22a60-f162-4ff3-8e30-292a3e977dd5_characterizeImage_946181_50.stderr
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e632e1391866b007af6e055510410d1f with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/478c8662-1163-4f4d-b66f-b6531e47f205_calibrate_946072_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/478c8662-1163-4f4d-b66f-b6531e47f205_calibrate_946072_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:40 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9aeed152a2363089a08e81f07922f45d with result from task 265
parsl.dataflow.dflow DEBUG: Task 265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 35537d171193c343269dcf8fe39f10c8
parsl.dataflow.memoization INFO: Task 266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 858
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2a561cb3-32a6-4e6e-a366-5a788414cd6d_calibrate_946074_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2a561cb3-32a6-4e6e-a366-5a788414cd6d_calibrate_946074_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/aa43edf6-8822-44d4-a47c-ac42d8982b45_characterizeImage_946074_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/aa43edf6-8822-44d4-a47c-ac42d8982b45_characterizeImage_946074_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:40 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfc73f0c6e69360c8c9189ceff56ced2 with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/83cbc48e-9cc4-493e-a529-f448f814ffc4_calibrate_946072_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/83cbc48e-9cc4-493e-a529-f448f814ffc4_calibrate_946072_40.stderr
parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 835595608ca5788686792b6315eaf39d with result from task 622
parsl.dataflow.dflow DEBUG: Task 622 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 8a0a65b62805cf45c1dab283aae4be91
parsl.dataflow.memoization INFO: Task 623 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088edd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 859
parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e9859b7f-84e4-45de-87a8-8a3a6fbea6d0_calibrate_946072_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e9859b7f-84e4-45de-87a8-8a3a6fbea6d0_calibrate_946072_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/31240a4e-ec49-4f3b-a539-a56472d32608_characterizeImage_946072_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/31240a4e-ec49-4f3b-a539-a56472d32608_characterizeImage_946072_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:41 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5de8fc63606772f3e2a78ee9d04ed46d with result from task 463
parsl.dataflow.dflow DEBUG: Task 463 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 464 has memoization hash d5ca859bc05eb03690031fad588e1afc
parsl.dataflow.memoization INFO: Task 464 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 860
parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3762998-7a2d-42bc-bae5-6f0f687563c3_calibrate_946181_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3762998-7a2d-42bc-bae5-6f0f687563c3_calibrate_946181_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3e23d4b3-954b-47cb-a762-20fb612548c2_characterizeImage_946181_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/3e23d4b3-954b-47cb-a762-20fb612548c2_characterizeImage_946181_57.stderr
parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 575c7fa8bf19a3ff8dcc434d4ad807cc with result from task 821
parsl.dataflow.dflow DEBUG: Task 821 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c1a1be0b-0fca-4cb3-a5d4-5fdaa72f8446_calibrate_946074_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c1a1be0b-0fca-4cb3-a5d4-5fdaa72f8446_calibrate_946074_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:41 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:42 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ceddc80ff1c779f966a0426d4c516bea with result from task 247
parsl.dataflow.dflow DEBUG: Task 247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 1174198c2d13206cb7412bdecf60bca6
parsl.dataflow.memoization INFO: Task 248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 861
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/1723df11-f141-41e9-adef-c4f8b71c38cb_calibrate_946072_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/1723df11-f141-41e9-adef-c4f8b71c38cb_calibrate_946072_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/acb69f8c-9e0d-49f0-b6e1-45c7b5acb8bf_characterizeImage_946072_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/acb69f8c-9e0d-49f0-b6e1-45c7b5acb8bf_characterizeImage_946072_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:42 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd8953c8d801f05e7d6dcf6abff8b4ad with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a7283126676a0ab6c6367741e62a0275
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 862
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3174f59-f0b9-4077-8c07-6a63b8362c21_calibrate_946073_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3174f59-f0b9-4077-8c07-6a63b8362c21_calibrate_946073_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b995bb2-e188-431a-ae07-1aa99731236b_characterizeImage_946073_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/1b995bb2-e188-431a-ae07-1aa99731236b_characterizeImage_946073_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:42 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fefc161d0551720011b1350ec7d15a4e with result from task 874
parsl.dataflow.dflow DEBUG: Task 874 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 1c4a5c73af2d84bb70aa3f82bc96fb22
parsl.dataflow.memoization INFO: Task 875 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 863
parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f6472e3f-bbc4-4653-9f89-56c5ee132616_calibrate_946181_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f6472e3f-bbc4-4653-9f89-56c5ee132616_calibrate_946181_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/108e573b-2521-4cd2-9fd9-8ec299c5b125_characterizeImage_946181_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/108e573b-2521-4cd2-9fd9-8ec299c5b125_characterizeImage_946181_22.stderr
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8cc7101394c0c5e00fcb1e77ec213371 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 021607d6f59899bc29db9722857bbb0d
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 864
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2d8e010e-e725-47fb-928b-cf336dd6f709_calibrate_946074_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2d8e010e-e725-47fb-928b-cf336dd6f709_calibrate_946074_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8b6f98c0-1ce0-4415-882a-c56ddd6dde4b_characterizeImage_946074_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8b6f98c0-1ce0-4415-882a-c56ddd6dde4b_characterizeImage_946074_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:43 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df97f203a999e0ae40b0d54e89cdb380 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 8b51d9bb8e820282ab328eaa3f9f7a6a
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 865
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d22cc4f4-e759-4aab-8506-8231996f9049_calibrate_946181_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d22cc4f4-e759-4aab-8506-8231996f9049_calibrate_946181_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d0261385-d7b5-4329-b587-6bf50d30054a_characterizeImage_946181_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d0261385-d7b5-4329-b587-6bf50d30054a_characterizeImage_946181_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:43 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03bc3e38ba82b253452d0e68f26bd91a with result from task 1105
parsl.dataflow.dflow DEBUG: Task 1105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash df44f6f5097e89cef5af941e5af41b93
parsl.dataflow.memoization INFO: Task 1106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 866
parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2f8e859-92a8-4329-b80b-34312a2d1543_calibrate_946182_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2f8e859-92a8-4329-b80b-34312a2d1543_calibrate_946182_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/20d39a7a-72d4-42fb-b170-0af7f3c398bf_characterizeImage_946182_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/20d39a7a-72d4-42fb-b170-0af7f3c398bf_characterizeImage_946182_5.stderr
parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a02357d7046e4466995ec91826b0f60 with result from task 493
parsl.dataflow.dflow DEBUG: Task 493 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 95ef636019694ea625deb2ce2b91c906
parsl.dataflow.memoization INFO: Task 494 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 867
parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/79f13d75-620e-49ef-bde4-26dac762c8ed_calibrate_946074_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/79f13d75-620e-49ef-bde4-26dac762c8ed_calibrate_946074_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5ea8cb2b-d830-454f-9282-39d125136784_characterizeImage_946074_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/5ea8cb2b-d830-454f-9282-39d125136784_characterizeImage_946074_51.stderr
parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 730609c62dc7b379ecdfb8b06ca747c2 with result from task 413
parsl.dataflow.dflow DEBUG: Task 413 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2ac1d876-262c-4775-9f6d-d137e90308aa_calibrate_946072_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2ac1d876-262c-4775-9f6d-d137e90308aa_calibrate_946072_47.stderr
parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 155928f0fc76de5eb6714ff12279e90c with result from task 469
parsl.dataflow.dflow DEBUG: Task 469 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 973cd8098a52fc6292109bafacb96d9e
parsl.dataflow.memoization INFO: Task 470 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 868
parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b85c0636-df72-4806-a8ad-d6918670f3b4_calibrate_946182_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b85c0636-df72-4806-a8ad-d6918670f3b4_calibrate_946182_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8998ca5c-b4c0-41f3-9649-8ffc637d5966_characterizeImage_946182_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8998ca5c-b4c0-41f3-9649-8ffc637d5966_characterizeImage_946182_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:44 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1024 try 1 failed
parsl.dataflow.dflow ERROR: Task 1024 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry d78583f1500fc98cf98036eff9d2a88d with result from task 1024
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1025 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1025 try 0 failed
parsl.dataflow.dflow INFO: Task 1025 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ff9eeb41-6d32-4810-9481-dd4d75240506_calibrate_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ff9eeb41-6d32-4810-9481-dd4d75240506_calibrate_946180_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/45cf0327-fb13-45a7-a524-a7feecb484a2_characterizeImage_946180_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:44 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 311e8db663f8c37dec9f058f373655b7 with result from task 236
parsl.dataflow.dflow DEBUG: Task 236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d2b34aa1-4bc7-4c7a-8420-17c00b674e0a_calibrate_946072_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d2b34aa1-4bc7-4c7a-8420-17c00b674e0a_calibrate_946072_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:45 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2798d6854fb236550087989817ae51f2 with result from task 722
parsl.dataflow.dflow DEBUG: Task 722 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93b22042-7099-4672-9117-69562dfb75e5_calibrate_946073_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93b22042-7099-4672-9117-69562dfb75e5_calibrate_946073_13.stderr
parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a202e71808bcb11935b4d4b026d2829 with result from task 568
parsl.dataflow.dflow DEBUG: Task 568 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 7e94d090b0a7a92bf5a1b0a5beec8a00
parsl.dataflow.memoization INFO: Task 569 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 869
parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8f0f71f8-674e-4125-af82-6e9677b41d94_calibrate_946072_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8f0f71f8-674e-4125-af82-6e9677b41d94_calibrate_946072_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/78ed2c3a-ed82-4f1b-9307-e8b9225ae4aa_characterizeImage_946072_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/78ed2c3a-ed82-4f1b-9307-e8b9225ae4aa_characterizeImage_946072_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:45 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c184be597c4d641c47a6d793ca418556 with result from task 631
parsl.dataflow.dflow DEBUG: Task 631 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 632 has memoization hash e5d3e69639578d7dd97d1fbc10558170
parsl.dataflow.memoization INFO: Task 632 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 870
parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/30b4d937-b39d-460a-a716-497f073d5f71_calibrate_946072_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/30b4d937-b39d-460a-a716-497f073d5f71_calibrate_946072_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4eeea007-f5fa-430d-ae82-52bd3f281d54_characterizeImage_946072_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4eeea007-f5fa-430d-ae82-52bd3f281d54_characterizeImage_946072_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:46 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc934904e8211c9beb8ca6aa3e2bef2a with result from task 616
parsl.dataflow.dflow DEBUG: Task 616 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 617 has memoization hash f62650595a2f385651871c08a0b28295
parsl.dataflow.memoization INFO: Task 617 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 871
parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/ffbbee0e-13c7-44d1-843a-100f08c2a465_calibrate_946073_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/ffbbee0e-13c7-44d1-843a-100f08c2a465_calibrate_946073_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7d42fd1b-dd37-4ffe-a5a5-89bd6dd9eead_characterizeImage_946073_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7d42fd1b-dd37-4ffe-a5a5-89bd6dd9eead_characterizeImage_946073_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:46 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11ee58d47f5c92cb7c81a6c522e27b2d with result from task 451
parsl.dataflow.dflow DEBUG: Task 451 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 452 has memoization hash b040e8a583310d132054acb7e658915f
parsl.dataflow.memoization INFO: Task 452 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 872
parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/060b33d4-1a74-424f-b03c-2faa4e91ee4b_calibrate_946181_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/060b33d4-1a74-424f-b03c-2faa4e91ee4b_calibrate_946181_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0f97984b-928e-4a1a-b60a-867b572eb572_characterizeImage_946181_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0f97984b-928e-4a1a-b60a-867b572eb572_characterizeImage_946181_53.stderr
parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 873e2b221879b656d468b1809d7e2429 with result from task 275
parsl.dataflow.dflow DEBUG: Task 275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1f380f6a-43af-4e59-9d1d-f7661bdfab50_calibrate_946073_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1f380f6a-43af-4e59-9d1d-f7661bdfab50_calibrate_946073_1.stderr
parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e78e5dfb34761251b6a28ed9ee6b65b with result from task 394
parsl.dataflow.dflow DEBUG: Task 394 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 12e1d09d3661440932f42de7174901d2
parsl.dataflow.memoization INFO: Task 395 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 873
parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c5f73a17-d7b8-45c2-94ca-a9c8db12c12a_calibrate_946181_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c5f73a17-d7b8-45c2-94ca-a9c8db12c12a_calibrate_946181_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/86cf2203-d675-471e-8f19-3135dbad3234_characterizeImage_946181_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/86cf2203-d675-471e-8f19-3135dbad3234_characterizeImage_946181_37.stderr
parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e47398088d0f2668dc6dc5774780657e with result from task 353
parsl.dataflow.dflow DEBUG: Task 353 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ace4bded-84af-4157-8d2c-c3d2db56ecca_calibrate_946072_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ace4bded-84af-4157-8d2c-c3d2db56ecca_calibrate_946072_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:46 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3cdecbaabe8959dc9be1b6f7ee1b80cb with result from task 724
parsl.dataflow.dflow DEBUG: Task 724 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 725 has memoization hash a3317fb8fea66e2f4b787f7e867c326b
parsl.dataflow.memoization INFO: Task 725 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 874
parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/988a0797-0f7e-4236-a8cc-3099bd5a49ab_calibrate_946181_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/988a0797-0f7e-4236-a8cc-3099bd5a49ab_calibrate_946181_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9cc3c20b-d748-41e9-8275-e17147b12621_characterizeImage_946181_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9cc3c20b-d748-41e9-8275-e17147b12621_characterizeImage_946181_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:47 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 491a8dbc1bed344780f8109dc4f1b0ee with result from task 457
parsl.dataflow.dflow DEBUG: Task 457 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 8b365617e81f210ff1c33ddea1495323
parsl.dataflow.memoization INFO: Task 458 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 875
parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ce9877f1-0e3b-4a48-97a9-857a92ae289f_calibrate_946180_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ce9877f1-0e3b-4a48-97a9-857a92ae289f_calibrate_946180_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5ba726dc-e9fa-4415-b39b-d4faab341260_characterizeImage_946180_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5ba726dc-e9fa-4415-b39b-d4faab341260_characterizeImage_946180_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:47 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18f75ebc8a58aaf9c8c70b80f8f9e56e with result from task 514
parsl.dataflow.dflow DEBUG: Task 514 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 515 has memoization hash b959283bda1e1980391f7e41152e5dd2
parsl.dataflow.memoization INFO: Task 515 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 876
parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d9c2e574-ef21-478f-a836-fb3921cbb956_calibrate_946072_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d9c2e574-ef21-478f-a836-fb3921cbb956_calibrate_946072_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9bbbe97c-eff9-4393-8c07-b48be2d482de_characterizeImage_946072_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9bbbe97c-eff9-4393-8c07-b48be2d482de_characterizeImage_946072_46.stderr
parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b5a2dda7dc6f0296616e9874fb38837 with result from task 529
parsl.dataflow.dflow DEBUG: Task 529 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 530 has memoization hash b4f694d6066b478303e84956dbcc10e9
parsl.dataflow.memoization INFO: Task 530 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 877
parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/53682e6c-22a9-4cf9-9710-7ff987536b46_calibrate_946073_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/53682e6c-22a9-4cf9-9710-7ff987536b46_calibrate_946073_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ca9bf1a9-12ed-4b27-90ba-2ec9566c9f8a_characterizeImage_946073_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ca9bf1a9-12ed-4b27-90ba-2ec9566c9f8a_characterizeImage_946073_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:48 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:48 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c0f5289d3f29afc94118bd775730fc7 with result from task 848
parsl.dataflow.dflow DEBUG: Task 848 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/20e219e4-5eeb-41df-93c9-5b1105f9438d_calibrate_946072_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/20e219e4-5eeb-41df-93c9-5b1105f9438d_calibrate_946072_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:49 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89e36bd77245a240e703bd7b706e5b04 with result from task 332
parsl.dataflow.dflow DEBUG: Task 332 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b31fcdd0-de93-4f5f-b06d-457df1214255_calibrate_946072_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b31fcdd0-de93-4f5f-b06d-457df1214255_calibrate_946072_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:49 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d8879e9591810c123f4f6aa64a77429 with result from task 716
parsl.dataflow.dflow DEBUG: Task 716 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/5bca9aa6-bbf7-4b51-9935-6e9cab1c2abe_calibrate_946073_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/5bca9aa6-bbf7-4b51-9935-6e9cab1c2abe_calibrate_946073_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:49 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09504d597b158032f7171d5a2325ddda with result from task 703
parsl.dataflow.dflow DEBUG: Task 703 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 6dad027af2becbd0ae157491bb7f960f
parsl.dataflow.memoization INFO: Task 704 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 878
parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9a568c54-ea78-4922-8518-314387c3c656_calibrate_946182_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9a568c54-ea78-4922-8518-314387c3c656_calibrate_946182_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/95aaf9e1-9584-4962-a29d-282677bca3f9_characterizeImage_946182_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/95aaf9e1-9584-4962-a29d-282677bca3f9_characterizeImage_946182_57.stderr
parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c99cccc3adc42bb93e5dbd40e5d304c6 with result from task 454
parsl.dataflow.dflow DEBUG: Task 454 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 29c0cbb4fbde79df4832238c7a67763d
parsl.dataflow.memoization INFO: Task 455 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 879
parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/49841fcd-5076-4f1f-bdeb-bb51f60ea2ef_calibrate_946181_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/49841fcd-5076-4f1f-bdeb-bb51f60ea2ef_calibrate_946181_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9398606f-9a50-4f31-818a-1680ef51858b_characterizeImage_946181_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/9398606f-9a50-4f31-818a-1680ef51858b_characterizeImage_946181_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:50 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc49950cc97ee6504203d566e5bb45ca with result from task 1100
parsl.dataflow.dflow DEBUG: Task 1100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/586d19fe-ad8e-4358-8176-030930a606e0_calibrate_946073_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/586d19fe-ad8e-4358-8176-030930a606e0_calibrate_946073_4.stderr
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7081284b8d14d7ac80a37d239a5d3fcf with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 7ca457ac6aff66d7d9a3affeb4691d8a
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f898a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 880
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/0a3bf06b-3a70-476f-b292-c2b95db5716a_calibrate_946072_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/0a3bf06b-3a70-476f-b292-c2b95db5716a_calibrate_946072_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5f99f95e-cbf8-4ed9-843d-276b5fc7767e_characterizeImage_946072_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/5f99f95e-cbf8-4ed9-843d-276b5fc7767e_characterizeImage_946072_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:50 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d76f60a0fbfb85a2e98ba8fd3467e1f with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2472aa6a-a767-4549-b9ac-f6d4c9f91a0e_calibrate_946072_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2472aa6a-a767-4549-b9ac-f6d4c9f91a0e_calibrate_946072_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:51 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7289cc915d8bfb42d88b1b025c8b64aa with result from task 271
parsl.dataflow.dflow DEBUG: Task 271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 18fe291ea7a568b1717db10dd7e6274c
parsl.dataflow.memoization INFO: Task 272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 881
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/9c199466-bbdf-4db0-9286-25cc8c36c549_calibrate_946180_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/9c199466-bbdf-4db0-9286-25cc8c36c549_calibrate_946180_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/52c2ea5c-ac0a-4eb0-bc8d-f6d67cfb877b_characterizeImage_946180_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/52c2ea5c-ac0a-4eb0-bc8d-f6d67cfb877b_characterizeImage_946180_5.stderr
parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff72189743838f7538fe44a88842df55 with result from task 203
parsl.dataflow.dflow DEBUG: Task 203 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/289546f5-5586-453b-b1c2-2b1a6ece237c_calibrate_946073_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/289546f5-5586-453b-b1c2-2b1a6ece237c_calibrate_946073_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:51 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 156bf146e23665379759e040d5feacf0 with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 5b74d83bd22ac8de88ca14001526696e
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 882
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/62f22ce3-54d4-47be-acee-fbab1bbb8fad_calibrate_946182_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/62f22ce3-54d4-47be-acee-fbab1bbb8fad_calibrate_946182_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/80236381-566b-4146-930a-5d657323e390_characterizeImage_946182_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/80236381-566b-4146-930a-5d657323e390_characterizeImage_946182_28.stderr
parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36825db7c77a6fb073cb0b38ea622562 with result from task 811
parsl.dataflow.dflow DEBUG: Task 811 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 431d0bc89efa28ecd82d26624b1e086a
parsl.dataflow.memoization INFO: Task 812 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 883
parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18d4836d-f5e8-4f51-86e3-cb4b54e3d68c_calibrate_946180_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18d4836d-f5e8-4f51-86e3-cb4b54e3d68c_calibrate_946180_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b4209ad1-aef6-41d7-bd15-2be7a4745bd0_characterizeImage_946180_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/b4209ad1-aef6-41d7-bd15-2be7a4745bd0_characterizeImage_946180_15.stderr
parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf5ebcfe6021e8664cdbaa146f153c0 with result from task 709
parsl.dataflow.dflow DEBUG: Task 709 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 8425528aed7a7b098bfb96e7daca2b57
parsl.dataflow.memoization INFO: Task 710 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 884
parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fa51ac9a-b855-40d2-b426-bd0c9f76e7d2_calibrate_946180_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fa51ac9a-b855-40d2-b426-bd0c9f76e7d2_calibrate_946180_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/542e2ab1-3a6a-4a41-899a-da680986d68e_characterizeImage_946180_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/542e2ab1-3a6a-4a41-899a-da680986d68e_characterizeImage_946180_4.stderr
parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cad697e6f25539291fad1f46fc7b91fb with result from task 356
parsl.dataflow.dflow DEBUG: Task 356 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5afac2b1-0e03-4066-b5c2-a28862b810f8_calibrate_946074_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5afac2b1-0e03-4066-b5c2-a28862b810f8_calibrate_946074_44.stderr
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f579f5d431bb4714c67907a557e522fa with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 04ae2711da8f7ae6f608a27c7684dad1
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 885
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4586958a-1864-4416-8bb0-25d79f7da970_calibrate_946073_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4586958a-1864-4416-8bb0-25d79f7da970_calibrate_946073_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7acd4-0e1b-4d21-a784-1d05d2c310ba_characterizeImage_946073_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e0b7acd4-0e1b-4d21-a784-1d05d2c310ba_characterizeImage_946073_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:51 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:52 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:52 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:53 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d21d91b14e77e07f24dac16a1329fe82 with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3ee67fcc-0c1a-4459-8d34-53c9c290eef5_calibrate_946072_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3ee67fcc-0c1a-4459-8d34-53c9c290eef5_calibrate_946072_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:53 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd1352a0f246414380e61155a10ebb1d with result from task 713
parsl.dataflow.dflow DEBUG: Task 713 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8ab0cfb4-a975-435e-a63d-df8cbd289db9_calibrate_946074_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8ab0cfb4-a975-435e-a63d-df8cbd289db9_calibrate_946074_21.stderr
parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 033da33d28c34395ddc1b62a469960ad with result from task 256
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:54 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow DEBUG: Task 256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 257 has memoization hash bfc1a5746149deaba6291130e6af561e
parsl.dataflow.memoization INFO: Task 257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 886
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/432484a1-14a3-433e-a20c-ddc0756f0e9e_calibrate_946180_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/432484a1-14a3-433e-a20c-ddc0756f0e9e_calibrate_946180_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ab4fe870-2d2a-49f6-a9bb-59e02d216ae2_characterizeImage_946180_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ab4fe870-2d2a-49f6-a9bb-59e02d216ae2_characterizeImage_946180_51.stderr
parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 220daf19531417b53011c7778b8ee1f7 with result from task 878
parsl.dataflow.dflow DEBUG: Task 878 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64a87fd3-6b56-4695-8f21-cdc97877b64c_calibrate_946073_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64a87fd3-6b56-4695-8f21-cdc97877b64c_calibrate_946073_31.stderr
parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3369ec1c9c65fb11aa7253799690dc12 with result from task 841
parsl.dataflow.dflow DEBUG: Task 841 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 18871936fccc40b3c723b270eaaa3c20
parsl.dataflow.memoization INFO: Task 842 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 887
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fc8da50f-6a55-4f1b-83ef-8216e83dbeb6_calibrate_946074_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fc8da50f-6a55-4f1b-83ef-8216e83dbeb6_calibrate_946074_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3517e8d9-fc98-43c4-8cef-4e7eec5dafc5_characterizeImage_946074_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3517e8d9-fc98-43c4-8cef-4e7eec5dafc5_characterizeImage_946074_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:54 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfeb060a50338ae21ca86561743bae94 with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c7e540ff-5e85-49ce-b71a-1c3519b32837_calibrate_946073_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c7e540ff-5e85-49ce-b71a-1c3519b32837_calibrate_946073_18.stderr
parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9951e0fd641f7e2d7caa6b67ce1a743e with result from task 670
parsl.dataflow.dflow DEBUG: Task 670 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 2a374bdcc92d9792a30c1b413ec4543f
parsl.dataflow.memoization INFO: Task 671 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 888
parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/29e96ea3-5939-4901-a8c1-ed9383837feb_calibrate_946182_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/29e96ea3-5939-4901-a8c1-ed9383837feb_calibrate_946182_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fc637ec4-c655-4ed1-9f92-7e9fc02ddc9e_characterizeImage_946182_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fc637ec4-c655-4ed1-9f92-7e9fc02ddc9e_characterizeImage_946182_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:54 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab9af1621c67ab41cf4e4fae5381019e with result from task 883
parsl.dataflow.dflow DEBUG: Task 883 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 23d8cf72ee3ec41be124cf4e0d49bdb2
parsl.dataflow.memoization INFO: Task 884 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 889
parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b1cc193b-7260-494a-a241-dae6971d26f5_calibrate_946180_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b1cc193b-7260-494a-a241-dae6971d26f5_calibrate_946180_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/df3164b7-0643-4cba-9027-7bedaa231722_characterizeImage_946180_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/df3164b7-0643-4cba-9027-7bedaa231722_characterizeImage_946180_49.stderr
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8041b28016a8d2f464761a3b12979e09 with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/639afbeb-6d5d-4457-ba27-a3cf495583a7_calibrate_946182_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/639afbeb-6d5d-4457-ba27-a3cf495583a7_calibrate_946182_60.stderr
parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3123b0a87476f8b42bf390bd6e8760b9 with result from task 313
parsl.dataflow.dflow DEBUG: Task 313 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 588f307652ad061e6ae3a71bdf52ab3c
parsl.dataflow.memoization INFO: Task 314 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 890
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0a2f8478-5a76-4af5-92ef-428cc7bf5441_calibrate_946074_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0a2f8478-5a76-4af5-92ef-428cc7bf5441_calibrate_946074_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/61babb40-3c60-48d4-888c-011075ee2598_characterizeImage_946074_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/61babb40-3c60-48d4-888c-011075ee2598_characterizeImage_946074_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:55 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c68a257cb11de08b9c71d80608eb5482 with result from task 916
parsl.dataflow.dflow DEBUG: Task 916 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 015a6b8584e256ff329ae274a3660d01
parsl.dataflow.memoization INFO: Task 917 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 891
parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1ca10c20-af16-4292-ac38-2b06ed3ee94e_calibrate_946181_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1ca10c20-af16-4292-ac38-2b06ed3ee94e_calibrate_946181_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d9a088b4-3821-47d6-9af8-0e2586f10546_characterizeImage_946181_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d9a088b4-3821-47d6-9af8-0e2586f10546_characterizeImage_946181_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:55 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:56 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc5943b8dedf5dc58ff84dc9192c5546 with result from task 577
parsl.dataflow.dflow DEBUG: Task 577 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 578 has memoization hash cbf884eae09f1a213841dffe230a7e00
parsl.dataflow.memoization INFO: Task 578 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 892
parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/897799e8-82cb-41dc-bccc-c132e3128dcc_calibrate_946180_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/897799e8-82cb-41dc-bccc-c132e3128dcc_calibrate_946180_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0d11567f-f42c-4428-81d7-f4c9fdaaf1a4_characterizeImage_946180_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0d11567f-f42c-4428-81d7-f4c9fdaaf1a4_characterizeImage_946180_27.stderr
parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 118f6639f62c1feabc8e737b9d1c21a4 with result from task 583
parsl.dataflow.dflow DEBUG: Task 583 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 1ef77c69ce350879bbce7d4770c83d9f
parsl.dataflow.memoization INFO: Task 584 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 893
parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/6853380c-03f4-4c8d-b860-0caf2a17aa10_calibrate_946182_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/6853380c-03f4-4c8d-b860-0caf2a17aa10_calibrate_946182_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7794f967-8a8f-49b8-9ac4-a3c25d3d1fbb_characterizeImage_946182_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/7794f967-8a8f-49b8-9ac4-a3c25d3d1fbb_characterizeImage_946182_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:56 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a4a89d82befa5605b68a4c99c253420 with result from task 571
parsl.dataflow.dflow DEBUG: Task 571 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 572 has memoization hash d545f2ca3ced70050fd8afb9c32687ae
parsl.dataflow.memoization INFO: Task 572 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 894
parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01d25c84-25e0-4e52-98d5-80d8f357719e_calibrate_946074_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01d25c84-25e0-4e52-98d5-80d8f357719e_calibrate_946074_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fd4a72a3-145f-4647-ad61-13e60a50d3d9_characterizeImage_946074_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/fd4a72a3-145f-4647-ad61-13e60a50d3d9_characterizeImage_946074_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:56 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4c15d70b8bf7ca4d093fb920f44bb8e with result from task 307
parsl.dataflow.dflow DEBUG: Task 307 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 308 has memoization hash f2614104c5c8d29c3919a50f56bd6fc0
parsl.dataflow.memoization INFO: Task 308 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 895
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6e97c7b-780b-4762-a19c-67dff364ec3d_calibrate_946074_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6e97c7b-780b-4762-a19c-67dff364ec3d_calibrate_946074_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7aaead84-de07-4093-ba17-6eb162a1b867_characterizeImage_946074_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7aaead84-de07-4093-ba17-6eb162a1b867_characterizeImage_946074_53.stderr
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02c26b40605cc6830891f42879119f9f with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash b3ce4f38c338f4f500b96dddeb1957ce
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 896
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/a1cb1b77-9700-467e-901d-72eeaa4b581c_calibrate_946072_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/a1cb1b77-9700-467e-901d-72eeaa4b581c_calibrate_946072_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/19fbbfca-f9b8-4dfa-aa70-5e452bae49b8_characterizeImage_946072_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/19fbbfca-f9b8-4dfa-aa70-5e452bae49b8_characterizeImage_946072_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:57 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97f6d2da9b8873bee3668b4b6a985faa with result from task 277
parsl.dataflow.dflow DEBUG: Task 277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 5b971996ca70c35d7438b8a1a2d9f4dd
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 897
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aed8ca06-1de3-4b27-988d-3b48aa2a3393_calibrate_946073_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aed8ca06-1de3-4b27-988d-3b48aa2a3393_calibrate_946073_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/97d29b98-683c-4966-9f83-240aac503d98_characterizeImage_946073_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/97d29b98-683c-4966-9f83-240aac503d98_characterizeImage_946073_49.stderr
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 479ad88babad5a1704451be14a21f586 with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash bed57f5360525db4ca16d5ab3467015c
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 898
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cc282278-09ea-40a6-a061-9857b55bdc26_calibrate_946073_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cc282278-09ea-40a6-a061-9857b55bdc26_calibrate_946073_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecf736aa-2147-4f58-a2e6-702ed9a863c6_characterizeImage_946073_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ecf736aa-2147-4f58-a2e6-702ed9a863c6_characterizeImage_946073_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:57 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa41f060355feec3f4b7b5a8b9ceff05 with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3ebee171-5337-4db1-a31b-72938b7e3e28_calibrate_946073_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3ebee171-5337-4db1-a31b-72938b7e3e28_calibrate_946073_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:58 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a62266926da326509618938edaf1984 with result from task 434
parsl.dataflow.dflow DEBUG: Task 434 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9a6a4fac-6200-4688-aac9-0bbdfc20e5de_calibrate_946073_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9a6a4fac-6200-4688-aac9-0bbdfc20e5de_calibrate_946073_19.stderr
parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cde93f3bfc937e2544de28d03d79a91 with result from task 230
parsl.dataflow.dflow DEBUG: Task 230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ba5007ab-3d3e-490c-95cd-97c6c4075039_calibrate_946181_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ba5007ab-3d3e-490c-95cd-97c6c4075039_calibrate_946181_32.stderr
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a601fe057b2501ecbc7fe7041274f3d8 with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/83cc5624-3706-4875-9c48-3b9c275ba9f8_calibrate_946182_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/83cc5624-3706-4875-9c48-3b9c275ba9f8_calibrate_946182_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:58 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:58 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 560fb2eff08dae1b6dae504304f2cf17 with result from task 637
parsl.dataflow.dflow DEBUG: Task 637 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 638 has memoization hash d10a86750913ac6a97d103954133a0a1
parsl.dataflow.memoization INFO: Task 638 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 899
parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/489dc325-fd6c-4139-8733-9ff230e16d70_calibrate_946073_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/489dc325-fd6c-4139-8733-9ff230e16d70_calibrate_946073_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9591292d-54a7-4efa-a76e-9f70dfd11824_characterizeImage_946073_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/9591292d-54a7-4efa-a76e-9f70dfd11824_characterizeImage_946073_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:59 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:37:59 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 240e9885389c3b954adea3a84ade579d with result from task 335
parsl.dataflow.dflow DEBUG: Task 335 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a3c749b2-ffa6-4c7c-960c-14d83e06c6ad_calibrate_946180_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a3c749b2-ffa6-4c7c-960c-14d83e06c6ad_calibrate_946180_31.stderr
parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a705677739dd7451d98c24a86a4a9807 with result from task 1051
parsl.dataflow.dflow DEBUG: Task 1051 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash c7fdfe496656d26727ca962ac386b3df
parsl.dataflow.memoization INFO: Task 1052 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 900
parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c03d0365-96f5-4410-9508-a6a107468bc0_calibrate_946181_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c03d0365-96f5-4410-9508-a6a107468bc0_calibrate_946181_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ff6c9446-9310-4245-b279-aa4d20e3e449_characterizeImage_946181_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ff6c9446-9310-4245-b279-aa4d20e3e449_characterizeImage_946181_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:00 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88912704e07d7ff20f00531476595ef0 with result from task 241
parsl.dataflow.dflow DEBUG: Task 241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 74a3318ded735d2e6f6ccb7a77c05942
parsl.dataflow.memoization INFO: Task 242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 901
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8255522f-081f-4a1f-8a98-8dec35dbffa5_calibrate_946182_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8255522f-081f-4a1f-8a98-8dec35dbffa5_calibrate_946182_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ec9c181b-6462-45ac-81f3-afbc5806fa36_characterizeImage_946182_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ec9c181b-6462-45ac-81f3-afbc5806fa36_characterizeImage_946182_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:00 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:00 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a65935f7988be0e066e62b3e47e682de with result from task 358
parsl.dataflow.dflow DEBUG: Task 358 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 20d4be282ee7da5c917250a26128336f
parsl.dataflow.memoization INFO: Task 359 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 902
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/75b8a4af-f988-4365-83f2-c8ab138c0fa3_calibrate_946072_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/75b8a4af-f988-4365-83f2-c8ab138c0fa3_calibrate_946072_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/f28ca535-ed4c-402b-9326-713376c40278_characterizeImage_946072_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/f28ca535-ed4c-402b-9326-713376c40278_characterizeImage_946072_38.stderr
parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b907664398dc68b3455caf4ed1eeed7d with result from task 901
parsl.dataflow.dflow DEBUG: Task 901 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 194b62e565289953f18e8049b3a62e9d
parsl.dataflow.memoization INFO: Task 902 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 903
parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f6bdc437-e632-43c0-8401-cdbd02d604e4_calibrate_946072_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f6bdc437-e632-43c0-8401-cdbd02d604e4_calibrate_946072_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/435feb3a-29c8-4937-82d4-891cf2704a47_characterizeImage_946072_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/435feb3a-29c8-4937-82d4-891cf2704a47_characterizeImage_946072_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:01 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2c8ed1c7bbe8111c6338af05b5b21d with result from task 377
parsl.dataflow.dflow DEBUG: Task 377 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/4e025109-feef-4cf7-a2af-a40a5b17d552_calibrate_946180_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/4e025109-feef-4cf7-a2af-a40a5b17d552_calibrate_946180_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:01 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06a0149a0e0a7dceabc4bbe1bc5dc0f2 with result from task 346
parsl.dataflow.dflow DEBUG: Task 346 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 347 has memoization hash c8b4130c5e99dbf627fc3fa2f21bf890
parsl.dataflow.memoization INFO: Task 347 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 904
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/57185630-5a52-47d8-9bbc-5ec56e9bb686_calibrate_946074_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/57185630-5a52-47d8-9bbc-5ec56e9bb686_calibrate_946074_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/b225f401-fd9e-4933-9006-bd1b0dc57d85_characterizeImage_946074_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/b225f401-fd9e-4933-9006-bd1b0dc57d85_characterizeImage_946074_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:02 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13c8b737d08675ab2f6c7cea731ccdd0 with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash f4418640ac4b0e79b2353f9d1c9f5977
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 905
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4f1c90c5-1609-4753-ac29-4cfc77286901_calibrate_946181_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4f1c90c5-1609-4753-ac29-4cfc77286901_calibrate_946181_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/59593040-90e1-4e7e-9d58-ce6b1a7ccc44_characterizeImage_946181_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/59593040-90e1-4e7e-9d58-ce6b1a7ccc44_characterizeImage_946181_58.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8318b522fd6685d391fb399ed5a4ec21 with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b4d30b18-0112-4e7e-9920-126421c5079a_calibrate_946073_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b4d30b18-0112-4e7e-9920-126421c5079a_calibrate_946073_48.stderr
parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec8abeea568b4f48f51c925f2e66fc9f with result from task 1111
parsl.dataflow.dflow DEBUG: Task 1111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 95cfa2196c020e04d4365f1a71a60edc
parsl.dataflow.memoization INFO: Task 1112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 906
parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0b6da01b-3278-4326-aaf4-3bdcae208a1f_calibrate_946074_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0b6da01b-3278-4326-aaf4-3bdcae208a1f_calibrate_946074_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/299a3ff9-a3f8-403e-95b6-d3db96ff404f_characterizeImage_946074_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/299a3ff9-a3f8-403e-95b6-d3db96ff404f_characterizeImage_946074_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:02 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e654fa0201f731e7c6e71a3bd70e1542 with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e2abe424-a741-4ca0-884c-f6b59171c422_calibrate_946072_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e2abe424-a741-4ca0-884c-f6b59171c422_calibrate_946072_55.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b0e6abff0b2e74244f35ef4ec944995 with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 73d86d12251351bd6b7986306767dd29
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 907
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/afaa6fae-c053-41ff-a19a-b3bee089c99e_calibrate_946072_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/afaa6fae-c053-41ff-a19a-b3bee089c99e_calibrate_946072_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/3578c3de-c1b0-46e9-9138-6f8c4a8c90a1_characterizeImage_946072_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/3578c3de-c1b0-46e9-9138-6f8c4a8c90a1_characterizeImage_946072_19.stderr
parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5dfc53ed9c7fa8230978592b0fcd8055 with result from task 625
parsl.dataflow.dflow DEBUG: Task 625 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 100b600d4983fb3dab12841b0020aed7
parsl.dataflow.memoization INFO: Task 626 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 908
parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6cd5551e-82b4-4ac2-93dd-833ff5bf11ce_calibrate_946072_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6cd5551e-82b4-4ac2-93dd-833ff5bf11ce_calibrate_946072_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d6c29223-5ccd-4cb9-b848-de00f17832f1_characterizeImage_946072_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d6c29223-5ccd-4cb9-b848-de00f17832f1_characterizeImage_946072_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:02 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84509bd9dad101e1d29e08ba125d0707 with result from task 967
parsl.dataflow.dflow DEBUG: Task 967 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 5d2c658d0ff684bab63fe179b3dd6430
parsl.dataflow.memoization INFO: Task 968 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 909
parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3753f76-3175-478d-aa36-e72031992688_calibrate_946073_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3753f76-3175-478d-aa36-e72031992688_calibrate_946073_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e3646bb9-1d4e-49bb-8053-bf3701e1a132_characterizeImage_946073_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e3646bb9-1d4e-49bb-8053-bf3701e1a132_characterizeImage_946073_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:03 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3bc7e944c7aa7d1b2b7fe967719bb8fb with result from task 386
parsl.dataflow.dflow DEBUG: Task 386 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/2188cf7d-2573-405a-a2d8-0b33f2830a63_calibrate_946073_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/2188cf7d-2573-405a-a2d8-0b33f2830a63_calibrate_946073_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:03 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 749f8639497d68a486157b1a31639f4b with result from task 1084
parsl.dataflow.dflow DEBUG: Task 1084 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 06d6d55feae053499aa3092608707f95
parsl.dataflow.memoization INFO: Task 1085 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 910
parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/03700e61-15af-4d96-9f6d-0d70bb34a1d0_calibrate_946072_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/03700e61-15af-4d96-9f6d-0d70bb34a1d0_calibrate_946072_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cbe95b10-0732-4b92-908b-2ea4b7d03bbd_characterizeImage_946072_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/cbe95b10-0732-4b92-908b-2ea4b7d03bbd_characterizeImage_946072_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:04 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9224c5eb454c698536b03094e9457056 with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash a71a8f1db67f8d54ceb3a0510f4fd533
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 911
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4da8fa31-7368-45ef-b9f6-2016b1ed6e01_calibrate_946074_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4da8fa31-7368-45ef-b9f6-2016b1ed6e01_calibrate_946074_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dbbde51f-57c4-4f0d-a6eb-d67da10e9e67_characterizeImage_946074_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dbbde51f-57c4-4f0d-a6eb-d67da10e9e67_characterizeImage_946074_10.stderr
parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e75e729f31232077740346993b78e279 with result from task 350
parsl.dataflow.dflow DEBUG: Task 350 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/54a733a9-179a-4192-8e47-ce4ecee2c084_calibrate_946180_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/54a733a9-179a-4192-8e47-ce4ecee2c084_calibrate_946180_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:04 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:04 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e3dd7afa62397f29a6bd4dceabb6a6a with result from task 844
parsl.dataflow.dflow DEBUG: Task 844 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 845 has memoization hash f9c0bd85e2fa044a525f196bf8bbfd19
parsl.dataflow.memoization INFO: Task 845 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 912
parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/19759d27-e92e-4a9f-a9c4-d935eb96c506_calibrate_946181_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/19759d27-e92e-4a9f-a9c4-d935eb96c506_calibrate_946181_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a64d37d6-c6cc-440f-afea-3c99994ad344_characterizeImage_946181_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a64d37d6-c6cc-440f-afea-3c99994ad344_characterizeImage_946181_27.stderr
parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 83a265744ac3a7e1dfa14d3c3d56c0be with result from task 488
parsl.dataflow.dflow DEBUG: Task 488 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/30a27325-f7e1-4094-b6de-31f2150a4246_calibrate_946074_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/30a27325-f7e1-4094-b6de-31f2150a4246_calibrate_946074_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:05 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b8f24f644278b4b83c56b7edf8e4ef4 with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d28d81ba-50d8-4e80-9878-1528e8c9f724_calibrate_946180_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d28d81ba-50d8-4e80-9878-1528e8c9f724_calibrate_946180_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:05 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:05 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 699a5ec418044513609ce686153275be with result from task 970
parsl.dataflow.dflow DEBUG: Task 970 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 971 has memoization hash b01af19a1f455f6f4f2e90e913e1f6e6
parsl.dataflow.memoization INFO: Task 971 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 913
parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/987ef3c6-326a-4b80-8f5c-6fd3073a56ce_calibrate_946072_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/987ef3c6-326a-4b80-8f5c-6fd3073a56ce_calibrate_946072_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/444ba73c-f344-4bc5-9f63-03c5deb4ef90_characterizeImage_946072_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/444ba73c-f344-4bc5-9f63-03c5deb4ef90_characterizeImage_946072_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:06 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7873059b6c8e4473a87b6b566e4820d7 with result from task 692
parsl.dataflow.dflow DEBUG: Task 692 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7ae2b82-e2f7-4269-9859-dd25959bea25_calibrate_946073_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7ae2b82-e2f7-4269-9859-dd25959bea25_calibrate_946073_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:06 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6ed06ea0117bc08ee79b76465c9895e with result from task 946
parsl.dataflow.dflow DEBUG: Task 946 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 947 has memoization hash cc6339189b5ff6a87ac01ce2f1719c50
parsl.dataflow.memoization INFO: Task 947 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 914
parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6a7daec-466f-44cb-983d-e08921cf337c_calibrate_946074_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6a7daec-466f-44cb-983d-e08921cf337c_calibrate_946074_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/86f3a392-c4eb-42d6-847d-3460291b2f7a_characterizeImage_946074_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/86f3a392-c4eb-42d6-847d-3460291b2f7a_characterizeImage_946074_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:06 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e3f3e00fa394d4a471a109bf161611f with result from task 1030
parsl.dataflow.dflow DEBUG: Task 1030 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 175b47381e49182cb85e820d943e3138
parsl.dataflow.memoization INFO: Task 1031 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 915
parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2fc0b3fe-0323-490f-8168-78cf2e55a744_calibrate_946182_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2fc0b3fe-0323-490f-8168-78cf2e55a744_calibrate_946182_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b1a62023-4b9f-48ae-b786-23c6dd482454_characterizeImage_946182_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b1a62023-4b9f-48ae-b786-23c6dd482454_characterizeImage_946182_55.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b01803ea529e8f891eaada3e4399e5e with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 5ba4b58d8b4b72de0d93aa500bd133b4
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f882c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 916
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b1911219-6ba6-45a1-99a0-2c6b1948b68d_calibrate_946073_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b1911219-6ba6-45a1-99a0-2c6b1948b68d_calibrate_946073_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d1eeb0db-1c51-46ec-9437-3081642a579a_characterizeImage_946073_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d1eeb0db-1c51-46ec-9437-3081642a579a_characterizeImage_946073_6.stderr
parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d752b1323ca2307d86c02a4cb2e00962 with result from task 964
parsl.dataflow.dflow DEBUG: Task 964 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 965 has memoization hash b0269037074bf2e7c702279a49412ccd
parsl.dataflow.memoization INFO: Task 965 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 917
parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b15535bd-a842-487c-9be2-3320e7ea0bcb_calibrate_946074_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b15535bd-a842-487c-9be2-3320e7ea0bcb_calibrate_946074_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f19878-7089-470e-b04f-d4a41f6cd43b_characterizeImage_946074_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f19878-7089-470e-b04f-d4a41f6cd43b_characterizeImage_946074_33.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 991 try 1 failed
parsl.dataflow.dflow ERROR: Task 991 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 95f2cab2908ff757b269ba6c1271bf58 with result from task 991
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 992 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 992 try 0 failed
parsl.dataflow.dflow INFO: Task 992 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ff290bf9-9808-4407-8f31-86dee684dfeb_calibrate_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ff290bf9-9808-4407-8f31-86dee684dfeb_calibrate_946072_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/e4f01e5d-9f59-4d97-969c-99bc524b6835_characterizeImage_946072_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:07 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 127adc499dc7d445ed19cb275c6220e6 with result from task 575
parsl.dataflow.dflow DEBUG: Task 575 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b01c19e6-e960-4d3d-9ea5-ee81582d989c_calibrate_946072_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b01c19e6-e960-4d3d-9ea5-ee81582d989c_calibrate_946072_59.stderr
parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84b1bd03b1aeae5dc5b464932e26e7fa with result from task 907
parsl.dataflow.dflow DEBUG: Task 907 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 908 has memoization hash ed359de278313e568e615113795826f0
parsl.dataflow.memoization INFO: Task 908 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 918
parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0a31e6ca-48fc-4598-9464-cd3bba240d1f_calibrate_946182_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0a31e6ca-48fc-4598-9464-cd3bba240d1f_calibrate_946182_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/57bac90d-b02b-43a4-b4d1-1094ad6acfd1_characterizeImage_946182_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/57bac90d-b02b-43a4-b4d1-1094ad6acfd1_characterizeImage_946182_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:07 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38c6a169ae2811920ef319f6aeeb09db with result from task 301
parsl.dataflow.dflow DEBUG: Task 301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 302 has memoization hash a650681b5eb40f552096f8134b69fe7c
parsl.dataflow.memoization INFO: Task 302 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 919
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a44829fc-dc99-4817-acf5-79f07f8c595d_calibrate_946182_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a44829fc-dc99-4817-acf5-79f07f8c595d_calibrate_946182_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/888d288f-dc6f-4cf3-95d3-1d7fe9202cbe_characterizeImage_946182_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/888d288f-dc6f-4cf3-95d3-1d7fe9202cbe_characterizeImage_946182_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:07 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3d5517014a3934b7e0fdb215a2120d9 with result from task 478
parsl.dataflow.dflow DEBUG: Task 478 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 2294204327c36c28a35b542e788e44e1
parsl.dataflow.memoization INFO: Task 479 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 920
parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3734002-702a-402d-9414-ef474961bc09_calibrate_946072_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3734002-702a-402d-9414-ef474961bc09_calibrate_946072_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c4a33271-a563-4db0-ad0f-509a2c294e27_characterizeImage_946072_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c4a33271-a563-4db0-ad0f-509a2c294e27_characterizeImage_946072_48.stderr
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4425cb2939407aa42e16611e603becfe with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash b0dee297654017dccfbe8656c40eea94
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 921
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d212850d-3c9c-4710-b719-be7a570a86d7_calibrate_946074_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d212850d-3c9c-4710-b719-be7a570a86d7_calibrate_946074_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dc3a5af1-fa70-4bfa-9773-400b0b779b49_characterizeImage_946074_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/dc3a5af1-fa70-4bfa-9773-400b0b779b49_characterizeImage_946074_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:08 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53ebcf5ba7ee5cac187fb2d2ec20f8a3 with result from task 880
parsl.dataflow.dflow DEBUG: Task 880 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 9829f5ade50bdbec319a577423f3b63a
parsl.dataflow.memoization INFO: Task 881 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 922
parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ac95aa6f-19ad-4942-93bc-74404920d9db_calibrate_946180_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ac95aa6f-19ad-4942-93bc-74404920d9db_calibrate_946180_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7e997f3d-5657-4727-afbd-f71b57fad166_characterizeImage_946180_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7e997f3d-5657-4727-afbd-f71b57fad166_characterizeImage_946180_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:08 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:08 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:08 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8562c02e152022d8f6f2750d9da3634 with result from task 581
parsl.dataflow.dflow DEBUG: Task 581 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/797ee7b9-b9d5-4a0b-bcc0-2eac949a52d6_calibrate_946072_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/797ee7b9-b9d5-4a0b-bcc0-2eac949a52d6_calibrate_946072_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:08 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2ffe02c4baf6938a35e3c05b9eefb31 with result from task 919
parsl.dataflow.dflow DEBUG: Task 919 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 5d824520f6218fef84a15ba1ad949e00
parsl.dataflow.memoization INFO: Task 920 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 923
parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e86340f8-c2eb-46fd-bd7f-284fefca75f8_calibrate_946074_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e86340f8-c2eb-46fd-bd7f-284fefca75f8_calibrate_946074_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3460f65e-6846-4277-9479-30cdcca10a8f_characterizeImage_946074_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3460f65e-6846-4277-9479-30cdcca10a8f_characterizeImage_946074_19.stderr
parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb4d8b1e43871b4c8c041d3047e6f695 with result from task 197
parsl.dataflow.dflow DEBUG: Task 197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/905474e2-e315-4ee3-b8ce-7d15f38f1637_calibrate_946073_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/905474e2-e315-4ee3-b8ce-7d15f38f1637_calibrate_946073_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:08 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53e4f30f93884335446e4550d7380b26 with result from task 295
parsl.dataflow.dflow DEBUG: Task 295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 049cc23340ad0768c944eb2c87b81084
parsl.dataflow.memoization INFO: Task 296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 924
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/be0735f0-c97c-4473-a404-f18078e3b80d_calibrate_946072_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/be0735f0-c97c-4473-a404-f18078e3b80d_calibrate_946072_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/05e17be8-4d28-452a-aca2-16a1103991e4_characterizeImage_946072_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/05e17be8-4d28-452a-aca2-16a1103991e4_characterizeImage_946072_41.stderr
parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac0b2d2aba104468b4a4b3bc3fef793 with result from task 832
parsl.dataflow.dflow DEBUG: Task 832 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 833 has memoization hash 6d6045993b2b2c6be27215101b9d821f
parsl.dataflow.memoization INFO: Task 833 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 925
parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cb66dfb-acdb-4f70-b1e7-74d0596c9afe_calibrate_946182_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cb66dfb-acdb-4f70-b1e7-74d0596c9afe_calibrate_946182_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8bc4e169-0e5a-489a-9f31-5e825ff2d3b6_characterizeImage_946182_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/8bc4e169-0e5a-489a-9f31-5e825ff2d3b6_characterizeImage_946182_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 82fc67c6cac65721de634d34a3a07144 with result from task 553
parsl.dataflow.dflow DEBUG: Task 553 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 1d2124559503ccacd4ea79b25188bcd8
parsl.dataflow.memoization INFO: Task 554 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 926
parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4d9cb6ca-b335-4d37-b003-e5f22269b9b6_calibrate_946074_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4d9cb6ca-b335-4d37-b003-e5f22269b9b6_calibrate_946074_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f514ca-8064-4c2c-958f-68ed4d775c12_characterizeImage_946074_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c9f514ca-8064-4c2c-958f-68ed4d775c12_characterizeImage_946074_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:09 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b47a276511ef58cd516d36f00e907304 with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 4f176528d8b44913c58fb4af766267ed
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 927
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c36ef7b7-0bbd-437d-a0d2-02e771d53d98_calibrate_946074_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c36ef7b7-0bbd-437d-a0d2-02e771d53d98_calibrate_946074_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/17638c39-c928-4320-98bc-9eb0cb7c3a50_characterizeImage_946074_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/17638c39-c928-4320-98bc-9eb0cb7c3a50_characterizeImage_946074_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f64296fb39e4e17871b0dd56df9f6c70 with result from task 484
parsl.dataflow.dflow DEBUG: Task 484 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 32164ea6a01a1072e09a4a631df9c06b
parsl.dataflow.memoization INFO: Task 485 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 928
parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/99772fea-2a4b-4d07-8df8-03098eb76fb6_calibrate_946073_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/99772fea-2a4b-4d07-8df8-03098eb76fb6_calibrate_946073_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/481e3cbf-6ff1-432a-9487-eb5e3d988c10_characterizeImage_946073_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/481e3cbf-6ff1-432a-9487-eb5e3d988c10_characterizeImage_946073_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 324
parsl.dataflow.strategy DEBUG: Executor multi has 324 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8edc284970f3d20078185b40589d27eb with result from task 973
parsl.dataflow.dflow DEBUG: Task 973 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 8dbc2a3729af60c4c50b91571b3f9193
parsl.dataflow.memoization INFO: Task 974 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 929
parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f7e633ed-d846-4dc0-a2c8-47210019f932_calibrate_946072_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f7e633ed-d846-4dc0-a2c8-47210019f932_calibrate_946072_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/196c7130-ceb9-4e00-bfb1-f64adabeee12_characterizeImage_946072_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/196c7130-ceb9-4e00-bfb1-f64adabeee12_characterizeImage_946072_53.stderr
parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9a3ac9e2e723f8adbdc116f3ca828c4 with result from task 700
parsl.dataflow.dflow DEBUG: Task 700 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 6e51899611fb21fbdc53be3150795255
parsl.dataflow.memoization INFO: Task 701 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 930
parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2caae7ac-3ce8-4b7d-965b-0b4c3805d08c_calibrate_946182_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2caae7ac-3ce8-4b7d-965b-0b4c3805d08c_calibrate_946182_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfa663d7-eb8c-40f2-b0d7-e27163075121_characterizeImage_946182_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfa663d7-eb8c-40f2-b0d7-e27163075121_characterizeImage_946182_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:10 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b784659c5fea48f1a197f4cef1612a2e with result from task 263
parsl.dataflow.dflow DEBUG: Task 263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/634567f4-b651-4e1d-a926-91b797abd8fe_calibrate_946074_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/634567f4-b651-4e1d-a926-91b797abd8fe_calibrate_946074_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 655dd6e68ebb0f0be9e406e08696e5ec with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 69e3a83c8a9abde7327bb8c274490472
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 931
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/74b8bf98-b91c-4261-8dc5-1dfa59e53813_calibrate_946072_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/74b8bf98-b91c-4261-8dc5-1dfa59e53813_calibrate_946072_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1f44b62d-2103-4839-906a-81c2284473ff_characterizeImage_946072_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/1f44b62d-2103-4839-906a-81c2284473ff_characterizeImage_946072_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:11 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 779c231920704f4c744697af55ff4984 with result from task 862
parsl.dataflow.dflow DEBUG: Task 862 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 863 has memoization hash d04c3c04643c0669162c4364fbc9412a
parsl.dataflow.memoization INFO: Task 863 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 932
parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/deaa91ea-8dad-4d4e-99f6-5e4f47f5ad6d_calibrate_946181_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/deaa91ea-8dad-4d4e-99f6-5e4f47f5ad6d_calibrate_946181_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d50206d7-28b5-47a1-9667-0b3ef71d57e3_characterizeImage_946181_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d50206d7-28b5-47a1-9667-0b3ef71d57e3_characterizeImage_946181_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34d9117833686c7eba63a0c0d6f3684b with result from task 227
parsl.dataflow.dflow DEBUG: Task 227 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c68a99fa-c607-4cca-b058-a64ee5002385_calibrate_946074_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c68a99fa-c607-4cca-b058-a64ee5002385_calibrate_946074_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc4c05f30028f4866c7ab665fe1e495 with result from task 284
parsl.dataflow.dflow DEBUG: Task 284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/cb021e2d-a4ef-4c41-973e-763d68eb8228_calibrate_946072_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/cb021e2d-a4ef-4c41-973e-763d68eb8228_calibrate_946072_8.stderr
parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1100cb8383d86df548b5dddb28031caf with result from task 1072
parsl.dataflow.dflow DEBUG: Task 1072 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash d798efaee12c56e249f70d1ae397aa8b
parsl.dataflow.memoization INFO: Task 1073 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 933
parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8920a4d0-8a2c-4ab4-90aa-e61d21d8f1d9_calibrate_946180_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8920a4d0-8a2c-4ab4-90aa-e61d21d8f1d9_calibrate_946180_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7846b11a-c2f3-4d69-8f00-376bfb855509_characterizeImage_946180_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/7846b11a-c2f3-4d69-8f00-376bfb855509_characterizeImage_946180_35.stderr
parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4654e187a1019bbf2ee9cc60941c0cac with result from task 958
parsl.dataflow.dflow DEBUG: Task 958 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 306e8c604ce78bd7dfbb2790e942d8ad
parsl.dataflow.memoization INFO: Task 959 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 934
parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/1331b609-d511-4661-9d12-c3492c42babe_calibrate_946180_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/1331b609-d511-4661-9d12-c3492c42babe_calibrate_946180_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2105b570-4538-4d9e-bb38-dd26f5601a7e_characterizeImage_946180_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2105b570-4538-4d9e-bb38-dd26f5601a7e_characterizeImage_946180_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1a16d838ed4d4c9f16ff76260413b51 with result from task 446
parsl.dataflow.dflow DEBUG: Task 446 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3e3d2c6a-87e8-46c4-a8ff-1f3ec20e991d_calibrate_946180_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3e3d2c6a-87e8-46c4-a8ff-1f3ec20e991d_calibrate_946180_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:12 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.memoization DEBUG: Storing app cache entry 021607d6f59899bc29db9722857bbb0d with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2d8e010e-e725-47fb-928b-cf336dd6f709_calibrate_946074_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2d8e010e-e725-47fb-928b-cf336dd6f709_calibrate_946074_31.stderr
parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1201055449e40df6ce3dba31568be22 with result from task 398
parsl.dataflow.dflow DEBUG: Task 398 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/a4370244-97c4-4bfe-bf86-d3b9ff654cdb_calibrate_946181_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/a4370244-97c4-4bfe-bf86-d3b9ff654cdb_calibrate_946181_51.stderr
parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d96d13698ed0d43ce8a8b0ec4130e5f3 with result from task 1094
parsl.dataflow.dflow DEBUG: Task 1094 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ffef99e6-3993-4211-bb64-09f442801d7b_calibrate_946072_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ffef99e6-3993-4211-bb64-09f442801d7b_calibrate_946072_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6dbf26e675fb703d35e59253ba826cd3 with result from task 889
parsl.dataflow.dflow DEBUG: Task 889 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 890 has memoization hash cd5a4c120c273c3729aef25dc763ee55
parsl.dataflow.memoization INFO: Task 890 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 935
parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/350f2cfb-45d6-41cc-b83e-845204144cfb_calibrate_946182_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/350f2cfb-45d6-41cc-b83e-845204144cfb_calibrate_946182_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/e991d81a-389f-4c09-a142-23a811a7aa8f_characterizeImage_946182_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/e991d81a-389f-4c09-a142-23a811a7aa8f_characterizeImage_946182_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:13 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 065f1351e5a5cf42095c0b19fa44a08c with result from task 562
parsl.dataflow.dflow DEBUG: Task 562 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 8ec93e9ada2630e81e4937590b0f95f8
parsl.dataflow.memoization INFO: Task 563 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 936
parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/588e707c-3d87-46a1-9b9e-a0d76ce608d2_calibrate_946074_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/588e707c-3d87-46a1-9b9e-a0d76ce608d2_calibrate_946074_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/367d2d53-9658-42e8-bd3b-00ce81ada50b_characterizeImage_946074_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/367d2d53-9658-42e8-bd3b-00ce81ada50b_characterizeImage_946074_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:14 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:14 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:14 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 82b69e9ced921447c47037903f7aa73f with result from task 200
parsl.dataflow.dflow DEBUG: Task 200 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9fc5c4b2-5048-440c-b8b7-fc438c34cbb9_calibrate_946181_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9fc5c4b2-5048-440c-b8b7-fc438c34cbb9_calibrate_946181_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:14 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 920a28fbf4bb82ed7c80c75a10a55e58 with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 7933cc12381c16e0418f7e83be65c46b
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 937
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c4db8bc4-387f-416e-81c5-1fa0c4ad1c93_calibrate_946074_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c4db8bc4-387f-416e-81c5-1fa0c4ad1c93_calibrate_946074_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ac29bff-c448-4ac9-a166-a4c31beb2b3b_characterizeImage_946074_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ac29bff-c448-4ac9-a166-a4c31beb2b3b_characterizeImage_946074_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:14 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57cbc3d037e15c5abee31d1ca4b1a1c6 with result from task 1102
parsl.dataflow.dflow DEBUG: Task 1102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash 8e1aff4edc07307a2bd0747bf3285378
parsl.dataflow.memoization INFO: Task 1103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 938
parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/be2dc455-f43b-4534-b8d3-a7c1b751e4b3_calibrate_946180_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/be2dc455-f43b-4534-b8d3-a7c1b751e4b3_calibrate_946180_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5b6c61f6-7156-4baf-9949-dc2a746f13bc_characterizeImage_946180_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5b6c61f6-7156-4baf-9949-dc2a746f13bc_characterizeImage_946180_42.stderr
parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd2a5280f254db98b87fbd75d9970e24 with result from task 389
parsl.dataflow.dflow DEBUG: Task 389 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2730465a-04d0-4a70-8dd5-4af2a5eb6d5c_calibrate_946072_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2730465a-04d0-4a70-8dd5-4af2a5eb6d5c_calibrate_946072_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:14 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0e9ed190a283bc15aaa5af3b797cddb with result from task 526
parsl.dataflow.dflow DEBUG: Task 526 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 527 has memoization hash ef420cdda25e96efd5c5459450e7a7d3
parsl.dataflow.memoization INFO: Task 527 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 939
parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4afd7cc1-2c4c-4c27-bcc4-4738f86e50d2_calibrate_946074_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4afd7cc1-2c4c-4c27-bcc4-4738f86e50d2_calibrate_946074_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/35e59a41-4f72-458e-88bf-65b950899c52_characterizeImage_946074_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/35e59a41-4f72-458e-88bf-65b950899c52_characterizeImage_946074_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:15 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:15 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b46dec850117b984aac14bc81dd4af28 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash b5b92cc854e36a16cc4a63886f3440f7
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 940
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/31dbcf4d-cb01-44c9-8a9f-b446637bac95_calibrate_946180_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/31dbcf4d-cb01-44c9-8a9f-b446637bac95_calibrate_946180_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3794f6fa-cc0a-48cc-96bd-95eddfd843c1_characterizeImage_946180_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3794f6fa-cc0a-48cc-96bd-95eddfd843c1_characterizeImage_946180_32.stderr
parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 924b0d4008c52cfbaf178dbb6b6beb33 with result from task 422
parsl.dataflow.dflow DEBUG: Task 422 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/9594e122-6c99-4591-a498-aead2fdfcf04_calibrate_946072_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/9594e122-6c99-4591-a498-aead2fdfcf04_calibrate_946072_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:15 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4edfa638a5ad64ab3eaa8f08278dc89 with result from task 310
parsl.dataflow.dflow DEBUG: Task 310 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 311 has memoization hash aa1bbee9da6b6f7a7b58e76a114859cc
parsl.dataflow.memoization INFO: Task 311 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 941
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/67711a9a-4742-49d8-be33-66afbd3c898f_calibrate_946181_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/67711a9a-4742-49d8-be33-66afbd3c898f_calibrate_946181_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5995a2ac-95f6-44f6-a0eb-f73ce9875bd2_characterizeImage_946181_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5995a2ac-95f6-44f6-a0eb-f73ce9875bd2_characterizeImage_946181_3.stderr
parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c508034f9489522dbbdfe265317e64c with result from task 319
parsl.dataflow.dflow DEBUG: Task 319 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 320 has memoization hash a9f39686d83645b226b33ecf7ffe6eac
parsl.dataflow.memoization INFO: Task 320 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 942
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/878e6bfb-62ca-4352-bcd9-4e19f3e3255a_calibrate_946182_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/878e6bfb-62ca-4352-bcd9-4e19f3e3255a_calibrate_946182_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/3bf4b117-1a32-4455-b84e-a947139ae66a_characterizeImage_946182_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/3bf4b117-1a32-4455-b84e-a947139ae66a_characterizeImage_946182_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:15 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dcbdca5a1b63e188215f50237dfed6e8 with result from task 1108
parsl.dataflow.dflow DEBUG: Task 1108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 22bbce3ba8df38b408c4827646bb3b79
parsl.dataflow.memoization INFO: Task 1109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 943
parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4560f55a-3ee9-4210-8e4f-55c057f060ac_calibrate_946074_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4560f55a-3ee9-4210-8e4f-55c057f060ac_calibrate_946074_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9e21cdcc-0b08-449b-bc21-3678f759d483_characterizeImage_946074_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9e21cdcc-0b08-449b-bc21-3678f759d483_characterizeImage_946074_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:15 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79ec4cb058243db3b9faa6afcd78881f with result from task 943
parsl.dataflow.dflow DEBUG: Task 943 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 299cbc1d254aa5db843646f24d4c3888
parsl.dataflow.memoization INFO: Task 944 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 944
parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/752361c9-eeab-44b8-b703-3ae3e964a122_calibrate_946074_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/752361c9-eeab-44b8-b703-3ae3e964a122_calibrate_946074_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de851926-21e1-4789-9e6f-63b089e344b6_characterizeImage_946074_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/de851926-21e1-4789-9e6f-63b089e344b6_characterizeImage_946074_42.stderr
parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd19cbfdeac1afddecabc2d93f1d8bbb with result from task 910
parsl.dataflow.dflow DEBUG: Task 910 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 0685ff97ec65a70b0f9f2d7e75d0acad
parsl.dataflow.memoization INFO: Task 911 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 945
parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5c73d9f1-44c4-41a2-b5b5-30704252ca96_calibrate_946182_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5c73d9f1-44c4-41a2-b5b5-30704252ca96_calibrate_946182_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5cae9711-b02e-4b09-aa52-42ce564dea92_characterizeImage_946182_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5cae9711-b02e-4b09-aa52-42ce564dea92_characterizeImage_946182_49.stderr
parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cf6da3659635e5b352f9a0d69408a30 with result from task 1097
parsl.dataflow.dflow DEBUG: Task 1097 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0538d2a6-a5d3-4486-98ae-1ae64177181a_calibrate_946182_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0538d2a6-a5d3-4486-98ae-1ae64177181a_calibrate_946182_27.stderr
parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae6c3d0b03d8ca1fe672cfc7570ea91 with result from task 250
parsl.dataflow.dflow DEBUG: Task 250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 251 has memoization hash cfa2a39852aef68d1329e977192b463d
parsl.dataflow.memoization INFO: Task 251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 946
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a0ececf2-e375-4a52-b7ff-f80b15a4dca7_calibrate_946073_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a0ececf2-e375-4a52-b7ff-f80b15a4dca7_calibrate_946073_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/291c352b-9f81-490a-a7d9-36ecbe428315_characterizeImage_946073_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/291c352b-9f81-490a-a7d9-36ecbe428315_characterizeImage_946073_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:16 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9948b53a8e961d484612b406b83db30c with result from task 362
parsl.dataflow.dflow DEBUG: Task 362 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b66cc311-6663-4bd2-a18b-4375947d7630_calibrate_946180_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b66cc311-6663-4bd2-a18b-4375947d7630_calibrate_946180_12.stderr
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce8465c298831705eee6329a5bd1f650 with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/284841a8-11dc-4b35-b99f-7560fc645db2_calibrate_946072_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/284841a8-11dc-4b35-b99f-7560fc645db2_calibrate_946072_52.stderr
parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7830e813e9715bd53440b56017cd6fc7 with result from task 1054
parsl.dataflow.dflow DEBUG: Task 1054 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash d2eade6c938151f583a0214a50a9f2b0
parsl.dataflow.memoization INFO: Task 1055 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 947
parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0b2d1b90-14fd-46ab-88b9-4d3b96ad6dec_calibrate_946182_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0b2d1b90-14fd-46ab-88b9-4d3b96ad6dec_calibrate_946182_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/49c7891e-fed8-4199-9a0e-cc42c76f81e4_characterizeImage_946182_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/49c7891e-fed8-4199-9a0e-cc42c76f81e4_characterizeImage_946182_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:16 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba30454913234955d9cc67b6a7259c77 with result from task 979
parsl.dataflow.dflow DEBUG: Task 979 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 2e8aa67185fd4a1b8f6841a2c62c39e2
parsl.dataflow.memoization INFO: Task 980 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 948
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/04629f76-7c03-4392-a99b-fd4abf2973ab_calibrate_946074_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/04629f76-7c03-4392-a99b-fd4abf2973ab_calibrate_946074_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8161566a-9ef4-4aa5-9a85-9e359ce65b8e_characterizeImage_946074_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8161566a-9ef4-4aa5-9a85-9e359ce65b8e_characterizeImage_946074_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:16 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:17 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7d7a01a72c6d8ab72dcbc2bc3ed4c7 with result from task 467
parsl.dataflow.dflow DEBUG: Task 467 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2fd81625-9859-4abd-a094-5d67ab913f03_calibrate_946072_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/2fd81625-9859-4abd-a094-5d67ab913f03_calibrate_946072_24.stderr
parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1bacd1f5cae6224ff808bb4f8fc08ec with result from task 1040
parsl.dataflow.dflow DEBUG: Task 1040 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/fec42c69-dea4-47e4-82a6-5a2624cde54e_calibrate_946181_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/fec42c69-dea4-47e4-82a6-5a2624cde54e_calibrate_946181_16.stderr
parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 073d26dd59677e10c28cdded741ed71a with result from task 221
parsl.dataflow.dflow DEBUG: Task 221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e4907077-9308-4e2f-a362-c1c89320c567_calibrate_946181_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e4907077-9308-4e2f-a362-c1c89320c567_calibrate_946181_35.stderr
parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6854ab86d62f9a3c99085ca30e2e6e36 with result from task 587
parsl.dataflow.dflow DEBUG: Task 587 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aa56f8bf-630e-42b5-9c93-a27bb3a74ce7_calibrate_946073_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aa56f8bf-630e-42b5-9c93-a27bb3a74ce7_calibrate_946073_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:17 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:18 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 596a8c3a5903b708f1a2c2239ae46793 with result from task 371
parsl.dataflow.dflow DEBUG: Task 371 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/9c491b54-a9ec-43c0-b661-39824d07a910_calibrate_946074_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/9c491b54-a9ec-43c0-b661-39824d07a910_calibrate_946074_26.stderr
parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7b9c87deaae3b76c56372e27122716e with result from task 476
parsl.dataflow.dflow DEBUG: Task 476 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/08469bbc-5403-4dca-a9df-9d179766c75c_calibrate_946074_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/08469bbc-5403-4dca-a9df-9d179766c75c_calibrate_946074_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:18 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f27581a6e3c98e812c078bd65509753 with result from task 218
parsl.dataflow.dflow DEBUG: Task 218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/36b96c52-c7d9-4119-a273-d33bef83809d_calibrate_946180_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/36b96c52-c7d9-4119-a273-d33bef83809d_calibrate_946180_39.stderr
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7283126676a0ab6c6367741e62a0275 with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3174f59-f0b9-4077-8c07-6a63b8362c21_calibrate_946073_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3174f59-f0b9-4077-8c07-6a63b8362c21_calibrate_946073_39.stderr
parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a0a65b62805cf45c1dab283aae4be91 with result from task 623
parsl.dataflow.dflow DEBUG: Task 623 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e9859b7f-84e4-45de-87a8-8a3a6fbea6d0_calibrate_946072_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/e9859b7f-84e4-45de-87a8-8a3a6fbea6d0_calibrate_946072_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:18 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:19 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:19 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5fdc61910e4b890d260bbd250b224b90 with result from task 628
parsl.dataflow.dflow DEBUG: Task 628 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 629 has memoization hash b1a06fc9bb97957d387c48ad3db1ea88
parsl.dataflow.memoization INFO: Task 629 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 949
parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5df4e049-6ad1-4099-9591-48d24d569664_calibrate_946181_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5df4e049-6ad1-4099-9591-48d24d569664_calibrate_946181_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/683c43ba-40a0-4835-bf9a-d79183ef8f01_characterizeImage_946181_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/683c43ba-40a0-4835-bf9a-d79183ef8f01_characterizeImage_946181_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:19 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5d3e69639578d7dd97d1fbc10558170 with result from task 632
parsl.dataflow.dflow DEBUG: Task 632 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/30b4d937-b39d-460a-a716-497f073d5f71_calibrate_946072_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/30b4d937-b39d-460a-a716-497f073d5f71_calibrate_946072_32.stderr
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81b9a62c098365c0998cb3106110aec7 with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6a0b1576-d3eb-4923-9727-4b8240e5905d_calibrate_946072_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6a0b1576-d3eb-4923-9727-4b8240e5905d_calibrate_946072_11.stderr
parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry beb2b6f591023cf56d9ae4ac1b39dc84 with result from task 952
parsl.dataflow.dflow DEBUG: Task 952 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 858c9ab7638285dec58910a7f6f9e194
parsl.dataflow.memoization INFO: Task 953 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 950
parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d561014a-854a-43e1-ae5f-6e061f1375c4_calibrate_946074_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d561014a-854a-43e1-ae5f-6e061f1375c4_calibrate_946074_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/10ee2fdf-17da-4e15-8457-3f3815057faf_characterizeImage_946074_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/10ee2fdf-17da-4e15-8457-3f3815057faf_characterizeImage_946074_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:20 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d89d5514b1695a172940d360a6cb6356 with result from task 940
parsl.dataflow.dflow DEBUG: Task 940 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 941 has memoization hash c3c57415374aa6fad6bb5103197a72dd
parsl.dataflow.memoization INFO: Task 941 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 951
parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ca6135f8-d725-452d-823d-054bf59804ee_calibrate_946180_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ca6135f8-d725-452d-823d-054bf59804ee_calibrate_946180_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/4159155a-d314-499d-aa80-34a8ca7c974f_characterizeImage_946180_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/4159155a-d314-499d-aa80-34a8ca7c974f_characterizeImage_946180_23.stderr
parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52db26d99fe15d08cb0670234b48bcf5 with result from task 269
parsl.dataflow.dflow DEBUG: Task 269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/87d1092b-ff25-401d-8213-3d809fc7e0e1_calibrate_946073_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/87d1092b-ff25-401d-8213-3d809fc7e0e1_calibrate_946073_5.stderr
parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41f6e4d423188d3fc0be5c9d49712f48 with result from task 209
parsl.dataflow.dflow DEBUG: Task 209 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/988b7b6c-9bfc-4afc-9c41-3af82784535b_calibrate_946182_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/988b7b6c-9bfc-4afc-9c41-3af82784535b_calibrate_946182_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:20 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22c3785f0f7c43a8c077a502adf0808c with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fee4ea8c-02fc-45ae-bd88-61b517d62c4f_calibrate_946074_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fee4ea8c-02fc-45ae-bd88-61b517d62c4f_calibrate_946074_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:20 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8ae463e5cfef9f5fcd056c20d08a4c with result from task 1000
parsl.dataflow.dflow DEBUG: Task 1000 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 7ae1c1e6e28cb5457028a6a6aeb4470d
parsl.dataflow.memoization INFO: Task 1001 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 952
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d618ff84-681e-4663-a641-de755752358c_calibrate_946073_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d618ff84-681e-4663-a641-de755752358c_calibrate_946073_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4400bafb-7bac-460d-b68e-a1b5d0ad7670_characterizeImage_946073_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/4400bafb-7bac-460d-b68e-a1b5d0ad7670_characterizeImage_946073_26.stderr
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d943c01ff235fbd9f4fb653671fb7b11 with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 2a8c06f25ada358c800fe2b8d62ae474
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f88c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 953
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9cd3467e-b10a-4df7-bd14-bc5048fdcdd0_calibrate_946181_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9cd3467e-b10a-4df7-bd14-bc5048fdcdd0_calibrate_946181_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/62e8d991-7177-4627-8304-930d48b3b945_characterizeImage_946181_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/62e8d991-7177-4627-8304-930d48b3b945_characterizeImage_946181_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:21 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b29d31e751de45428f2b3a8cef7151f with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c3b5377d-b626-4019-90a5-01c8be216821_calibrate_946181_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c3b5377d-b626-4019-90a5-01c8be216821_calibrate_946181_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:21 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36b2470f087edb0e12e4d94893b6d049 with result from task 499
parsl.dataflow.dflow DEBUG: Task 499 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 500 has memoization hash c351699d96b455f2ca717eb565dff491
parsl.dataflow.memoization INFO: Task 500 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 954
parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a0953b81-dc92-45cb-8ac1-dc8b9a55696b_calibrate_946180_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a0953b81-dc92-45cb-8ac1-dc8b9a55696b_calibrate_946180_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2abf49d3-dc0a-4529-83a6-b7379aa605a5_characterizeImage_946180_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/2abf49d3-dc0a-4529-83a6-b7379aa605a5_characterizeImage_946180_22.stderr
parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0f72a0946deb87c68e52daf6855f2a9 with result from task 517
parsl.dataflow.dflow DEBUG: Task 517 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 523b59f4d025193dc802b85c7cf83668
parsl.dataflow.memoization INFO: Task 518 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a76f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 955
parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b30ffcaa-230e-4390-bae0-571ffaec8ff5_calibrate_946074_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b30ffcaa-230e-4390-bae0-571ffaec8ff5_calibrate_946074_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c5cdd94a-7439-4651-a91d-121d2c934afa_characterizeImage_946074_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/c5cdd94a-7439-4651-a91d-121d2c934afa_characterizeImage_946074_3.stderr
parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c809592a08e86bd3ba89f8b0cc3d2196 with result from task 323
parsl.dataflow.dflow DEBUG: Task 323 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fafca06d-e7ef-490c-8135-2b1113ab22e2_calibrate_946180_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fafca06d-e7ef-490c-8135-2b1113ab22e2_calibrate_946180_3.stderr
parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 866278259ae65d5b22de3810ff5c6857 with result from task 835
parsl.dataflow.dflow DEBUG: Task 835 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 957a67948448dd2a2d9326ea15697c37
parsl.dataflow.memoization INFO: Task 836 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 956
parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d7890866-5e0e-47a8-9b5a-1c6698bf09c9_calibrate_946074_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d7890866-5e0e-47a8-9b5a-1c6698bf09c9_calibrate_946074_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/f1f0f7b1-5ec6-471a-8ec5-066628a3d5de_characterizeImage_946074_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/f1f0f7b1-5ec6-471a-8ec5-066628a3d5de_characterizeImage_946074_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:21 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f418e2bd8881aa66c473cad13bf04eec with result from task 685
parsl.dataflow.dflow DEBUG: Task 685 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 0b68298a38d1fe94c24a45cbe0c169ae
parsl.dataflow.memoization INFO: Task 686 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 957
parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8d7cceae-0e24-4c0e-b812-991354e3b417_calibrate_946072_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8d7cceae-0e24-4c0e-b812-991354e3b417_calibrate_946072_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/be98b632-499f-4cfa-99c4-1ccb8472bfae_characterizeImage_946072_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/be98b632-499f-4cfa-99c4-1ccb8472bfae_characterizeImage_946072_27.stderr
parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecd412fee0560e965887de5fb2720278 with result from task 391
parsl.dataflow.dflow DEBUG: Task 391 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 392 has memoization hash d5f8c2c2a141de4ff9e9907725a56dfc
parsl.dataflow.memoization INFO: Task 392 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 958
parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/5f032f64-4247-472f-819b-bb41b788fc40_calibrate_946072_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/5f032f64-4247-472f-819b-bb41b788fc40_calibrate_946072_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/b03e2b6a-49df-4351-aecb-d9215e1ebc1a_characterizeImage_946072_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/b03e2b6a-49df-4351-aecb-d9215e1ebc1a_characterizeImage_946072_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:21 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88dcfaaab25c1aecd38aa0985125a364 with result from task 206
parsl.dataflow.dflow DEBUG: Task 206 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ddbae87f-6fc7-4dc3-a25c-b5ca3ba1f5e2_calibrate_946180_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ddbae87f-6fc7-4dc3-a25c-b5ca3ba1f5e2_calibrate_946180_59.stderr
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 341701b9a22f32d158764d3771495f95 with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 43a44a1f3f063ea365f9f43c9228f2e3
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 959
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc06821a-ef51-4654-9f13-38876dd495ef_calibrate_946180_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc06821a-ef51-4654-9f13-38876dd495ef_calibrate_946180_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8f224ac8-7b26-45ef-a478-f51b318fb7ff_characterizeImage_946180_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8f224ac8-7b26-45ef-a478-f51b318fb7ff_characterizeImage_946180_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:21 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 960a1d08e8e3ff0940a38a3db43ac935 with result from task 544
parsl.dataflow.dflow DEBUG: Task 544 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 545 has memoization hash d84b5640ad9a4e759949d5dcbf279d2d
parsl.dataflow.memoization INFO: Task 545 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 960
parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a163f383-a097-425e-ad60-32917cfdb990_calibrate_946180_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a163f383-a097-425e-ad60-32917cfdb990_calibrate_946180_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fc27e22a-ae6c-4ebd-b334-1d8cd9db0366_characterizeImage_946180_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fc27e22a-ae6c-4ebd-b334-1d8cd9db0366_characterizeImage_946180_1.stderr
parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f62650595a2f385651871c08a0b28295 with result from task 617
parsl.dataflow.dflow DEBUG: Task 617 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/ffbbee0e-13c7-44d1-843a-100f08c2a465_calibrate_946073_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/ffbbee0e-13c7-44d1-843a-100f08c2a465_calibrate_946073_47.stderr
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a360d24acadd11b782aae373cc80a9fe with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/86b9dd13-be38-4ca8-9acc-52d4f5e1799c_calibrate_946074_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/86b9dd13-be38-4ca8-9acc-52d4f5e1799c_calibrate_946074_48.stderr
parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.dataflow.memoization DEBUG: Storing app cache entry e6036a015d6c495ddb3a22961fb73f5a with result from task 698
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 698 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/52609732-02fe-41d5-b6b4-8772c9654b51_calibrate_946182_20.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/52609732-02fe-41d5-b6b4-8772c9654b51_calibrate_946182_20.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:22 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d070af9c8d546a07d88bad62aea0081a with result from task 818
parsl.dataflow.dflow DEBUG: Task 818 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/461368d5-440d-4179-a78d-d2b83563179c_calibrate_946073_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/461368d5-440d-4179-a78d-d2b83563179c_calibrate_946073_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:22 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:22 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 728df41a0f1721fc89c754ccb3fb7567 with result from task 490
parsl.dataflow.dflow DEBUG: Task 490 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 491 has memoization hash a719e0a3393c403753ab8d20e472d495
parsl.dataflow.memoization INFO: Task 491 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 961
parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0016d22e-b80b-40a0-a952-8049f2a2855a_calibrate_946181_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0016d22e-b80b-40a0-a952-8049f2a2855a_calibrate_946181_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0d45ff8a-1a35-4f2c-ab94-01cbc5cc4d22_characterizeImage_946181_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0d45ff8a-1a35-4f2c-ab94-01cbc5cc4d22_characterizeImage_946181_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:22 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1dc3edd88df69bf04b980e6d29c3191e with result from task 508
parsl.dataflow.dflow DEBUG: Task 508 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 88e90dde7bc3b62e9f33b25118a9afca
parsl.dataflow.memoization INFO: Task 509 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 962
parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0ede9ad5-a2a4-41e9-90d1-705944fe99d8_calibrate_946182_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0ede9ad5-a2a4-41e9-90d1-705944fe99d8_calibrate_946182_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ed9d1294-2e41-4959-b1f6-9705206f78a9_characterizeImage_946182_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/ed9d1294-2e41-4959-b1f6-9705206f78a9_characterizeImage_946182_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:22 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c12268d4f7c7c34978fc12dd34c70f33 with result from task 365
parsl.dataflow.dflow DEBUG: Task 365 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c77c0ec4-e10e-404e-b4fb-2f99d9b7b7f6_calibrate_946180_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c77c0ec4-e10e-404e-b4fb-2f99d9b7b7f6_calibrate_946180_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:22 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc7fee1123aa1606fc4bab7ad4de8fe with result from task 829
parsl.dataflow.dflow DEBUG: Task 829 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 830 has memoization hash da6ac0edf8c814d9298374c572c52ea0
parsl.dataflow.memoization INFO: Task 830 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 963
parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d80ec8ce-f312-4ffc-b4f3-9bf246fa3f8c_calibrate_946180_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d80ec8ce-f312-4ffc-b4f3-9bf246fa3f8c_calibrate_946180_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6910a32e-99e1-4300-bcfd-e31a4e5600c6_characterizeImage_946180_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/6910a32e-99e1-4300-bcfd-e31a4e5600c6_characterizeImage_946180_50.stderr
parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8242cff426e524754e3d225c488f1531 with result from task 404
parsl.dataflow.dflow DEBUG: Task 404 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0db56701-0a64-4eb0-887f-894833ff87e8_calibrate_946074_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0db56701-0a64-4eb0-887f-894833ff87e8_calibrate_946074_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c5d29faea73e66d40ed9c4898d7a029 with result from task 316
parsl.dataflow.dflow DEBUG: Task 316 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 2036fd0e1a4b228e29f019a9d1b710fb
parsl.dataflow.memoization INFO: Task 317 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 964
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c7ec35ce-273c-4dc1-8dea-fe7f289a6abf_calibrate_946180_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c7ec35ce-273c-4dc1-8dea-fe7f289a6abf_calibrate_946180_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/997fb956-ed0c-456e-becc-3aa0c88201fd_characterizeImage_946180_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/997fb956-ed0c-456e-becc-3aa0c88201fd_characterizeImage_946180_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7173a9512812d7ce587a9d50fba6b700 with result from task 682
parsl.dataflow.dflow DEBUG: Task 682 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 683 has memoization hash e1e5d5c0398d221713eb6b3ac08d4b99
parsl.dataflow.memoization INFO: Task 683 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 965
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/66d99a80-20f2-4bee-9183-bb154ea58320_calibrate_946072_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/66d99a80-20f2-4bee-9183-bb154ea58320_calibrate_946072_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8c136890-3413-4bfe-a8fd-c6ba6fb6c3f9_characterizeImage_946072_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8c136890-3413-4bfe-a8fd-c6ba6fb6c3f9_characterizeImage_946072_22.stderr
parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9db19a50defcd33efd43d7b5885e584 with result from task 418
parsl.dataflow.dflow DEBUG: Task 418 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 419 has memoization hash a4e9561e8fb858dde147c4b1651ece6d
parsl.dataflow.memoization INFO: Task 419 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 966
parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/361e4c74-c378-4aa7-98ff-521d1bd64e87_calibrate_946072_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/361e4c74-c378-4aa7-98ff-521d1bd64e87_calibrate_946072_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8bb9db62-4eca-459e-b094-97eb8e1c0ed7_characterizeImage_946072_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/8bb9db62-4eca-459e-b094-97eb8e1c0ed7_characterizeImage_946072_54.stderr
parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccd9e429f0b45929ab262f230c4c46e8 with result from task 838
parsl.dataflow.dflow DEBUG: Task 838 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 35fb18251901972b1a52689ac3b8adfe
parsl.dataflow.memoization INFO: Task 839 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 967
parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/da01e288-092c-449d-a722-fe4381713a43_calibrate_946182_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/da01e288-092c-449d-a722-fe4381713a43_calibrate_946182_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/560d8e4c-61f4-4456-ad7c-c398cf0457aa_characterizeImage_946182_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/560d8e4c-61f4-4456-ad7c-c398cf0457aa_characterizeImage_946182_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:23 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cadcd7409b293d6f41cd20ddcbbe9a87 with result from task 443
parsl.dataflow.dflow DEBUG: Task 443 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24829295-9a55-49ce-93a6-6a076a962705_calibrate_946181_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24829295-9a55-49ce-93a6-6a076a962705_calibrate_946181_60.stderr
parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f028390d9c4a5ca157d6c547795ceeae with result from task 556
parsl.dataflow.dflow DEBUG: Task 556 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 68d22eddcf43952df068c02117b1e079
parsl.dataflow.memoization INFO: Task 557 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 968
parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8b8b5232-6d9d-49ec-84de-4771588553f4_calibrate_946182_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8b8b5232-6d9d-49ec-84de-4771588553f4_calibrate_946182_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c7fd7132-c5c4-4e6c-9978-ff3fe2997010_characterizeImage_946182_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c7fd7132-c5c4-4e6c-9978-ff3fe2997010_characterizeImage_946182_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 251c79f8bef17953c3a4a1b48e466225 with result from task 619
parsl.dataflow.dflow DEBUG: Task 619 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 9a1b66c91f5c9eacc260d192518d7747
parsl.dataflow.memoization INFO: Task 620 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 969
parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/479f0b6d-99f7-40ec-9ad9-f9e09f472b76_calibrate_946182_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/479f0b6d-99f7-40ec-9ad9-f9e09f472b76_calibrate_946182_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/75a864b3-60f0-4501-8883-26fad1e48bd6_characterizeImage_946182_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/75a864b3-60f0-4501-8883-26fad1e48bd6_characterizeImage_946182_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b3d1a65f460f2af1c216709b1c83aea with result from task 815
parsl.dataflow.dflow DEBUG: Task 815 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2f1cf6ea-94c1-47d6-a986-379cf21ea46e_calibrate_946074_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2f1cf6ea-94c1-47d6-a986-379cf21ea46e_calibrate_946074_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f180a03d1763b08c822a70ba23024f4 with result from task 511
parsl.dataflow.dflow DEBUG: Task 511 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 385716257c7bdbe8df0534e6439cb947
parsl.dataflow.memoization INFO: Task 512 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 970
parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7a954a6-bb06-4055-a7d9-4fe3d4a0b712_calibrate_946073_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7a954a6-bb06-4055-a7d9-4fe3d4a0b712_calibrate_946073_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/fdbd1016-9fce-4141-a81a-81360ddb9a41_characterizeImage_946073_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/fdbd1016-9fce-4141-a81a-81360ddb9a41_characterizeImage_946073_17.stderr
parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 302a71c2a2e3a648fa6959516286fd14 with result from task 649
parsl.dataflow.dflow DEBUG: Task 649 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 9ea55bd1ec95dd2ee4c2724bbc4a41d1
parsl.dataflow.memoization INFO: Task 650 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 971
parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7cb7f79a-ffd9-4ad4-8b96-d33533761b7e_calibrate_946072_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7cb7f79a-ffd9-4ad4-8b96-d33533761b7e_calibrate_946072_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/fc5869a3-9f3e-4bc5-b03e-08cce6d89c24_characterizeImage_946072_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/fc5869a3-9f3e-4bc5-b03e-08cce6d89c24_characterizeImage_946072_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:24 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:25 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:26 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:26 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:26 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 020961893ee8a2bd61f7a3be1ac98236 with result from task 233
parsl.dataflow.dflow DEBUG: Task 233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/73581bf2-4f2e-467b-b9b3-b2ab20ee4ffd_calibrate_946182_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/73581bf2-4f2e-467b-b9b3-b2ab20ee4ffd_calibrate_946182_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:26 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:26 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 39cbab66dd4cc9d7c8b634f75bed6b45 with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a3bc34f7-97ec-4c11-be45-dcfeccafd3a3_calibrate_946073_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a3bc34f7-97ec-4c11-be45-dcfeccafd3a3_calibrate_946073_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:26 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 284
parsl.dataflow.strategy DEBUG: Executor multi has 284 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 284
parsl.dataflow.strategy DEBUG: Executor multi has 284 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ea9840f3aefbe88c8ad885a93997988 with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/42a75377-42a2-4986-ad03-59e0db69d793_calibrate_946182_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/42a75377-42a2-4986-ad03-59e0db69d793_calibrate_946182_43.stderr
parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 29bd8d15a422f5edaea1f9ca2a6c744e with result from task 535
parsl.dataflow.dflow DEBUG: Task 535 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 536 has memoization hash c24bf40c47b3c113f227c922d27aceca
parsl.dataflow.memoization INFO: Task 536 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 972
parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b39beabf-b6c8-4cff-acd1-f39fbf7a0dee_calibrate_946074_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b39beabf-b6c8-4cff-acd1-f39fbf7a0dee_calibrate_946074_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/a06f6976-ec38-4626-ac60-54bc5b825153_characterizeImage_946074_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/a06f6976-ec38-4626-ac60-54bc5b825153_characterizeImage_946074_41.stderr
parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 39f7ee7ca594eb1d8120aed1c8ebf6db with result from task 823
parsl.dataflow.dflow DEBUG: Task 823 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 1c3c2ef5fb808af988933a5e95b85a06
parsl.dataflow.memoization INFO: Task 824 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 973
parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/135d6146-f86b-452b-a29f-eb73ee15e4a5_calibrate_946180_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/135d6146-f86b-452b-a29f-eb73ee15e4a5_calibrate_946180_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/cd9fd5ba-f5ae-4559-992e-795aca3b645e_characterizeImage_946180_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/cd9fd5ba-f5ae-4559-992e-795aca3b645e_characterizeImage_946180_47.stderr
parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53c2a1badcb7e114c042d22825af55fb with result from task 667
parsl.dataflow.dflow DEBUG: Task 667 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 668 has memoization hash c14b01b44934a9347950f1224f184c89
parsl.dataflow.memoization INFO: Task 668 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 974
parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7831dcaa-4a9c-40f9-88c9-826a982707e4_calibrate_946072_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7831dcaa-4a9c-40f9-88c9-826a982707e4_calibrate_946072_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/577ed6c1-358b-4b7a-b5c1-c92e61ae8798_characterizeImage_946072_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/577ed6c1-358b-4b7a-b5c1-c92e61ae8798_characterizeImage_946072_17.stderr
parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae2534b71b8c398b687324ca8e928ac6 with result from task 719
parsl.dataflow.dflow DEBUG: Task 719 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/41cba618-a9ac-4c98-8227-4154630d5b08_calibrate_946072_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/41cba618-a9ac-4c98-8227-4154630d5b08_calibrate_946072_62.stderr
parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e3d5940fbf8da867a2a9c2cb4a0a05af with result from task 653
parsl.dataflow.dflow DEBUG: Task 653 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/638353bf-fbfa-40a6-a91a-5cb8d3aaf19f_calibrate_946074_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/638353bf-fbfa-40a6-a91a-5cb8d3aaf19f_calibrate_946074_59.stderr
parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a5a60345d9adce287849f106fc47bce with result from task 590
parsl.dataflow.dflow DEBUG: Task 590 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ff4d4748-bf41-47eb-8c64-f6b57497f039_calibrate_946072_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ff4d4748-bf41-47eb-8c64-f6b57497f039_calibrate_946072_39.stderr
parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b389b039cfeb16b0e1fa5d59156c95b5 with result from task 688
parsl.dataflow.dflow DEBUG: Task 688 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 4f8baa323e81eb5c626120e45630507a
parsl.dataflow.memoization INFO: Task 689 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308936200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 975
parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c0982490-7c40-494f-9fec-971e865efc2e_calibrate_946180_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c0982490-7c40-494f-9fec-971e865efc2e_calibrate_946180_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/bc018c03-a417-4862-b532-1c7e8f2ec9d5_characterizeImage_946180_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/bc018c03-a417-4862-b532-1c7e8f2ec9d5_characterizeImage_946180_26.stderr
parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35537d171193c343269dcf8fe39f10c8 with result from task 266
parsl.dataflow.dflow DEBUG: Task 266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2a561cb3-32a6-4e6e-a366-5a788414cd6d_calibrate_946074_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/2a561cb3-32a6-4e6e-a366-5a788414cd6d_calibrate_946074_6.stderr
parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e798cf8056ef99def8787d62ef757ec with result from task 1060
parsl.dataflow.dflow DEBUG: Task 1060 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash a9372d35ac69794534f2dceb5b8afbdc
parsl.dataflow.memoization INFO: Task 1061 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 976
parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0379e2ae-a736-4a23-8e1b-fd501ae4eaad_calibrate_946181_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0379e2ae-a736-4a23-8e1b-fd501ae4eaad_calibrate_946181_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7295d796-0203-44c6-8275-d20462ef5726_characterizeImage_946181_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7295d796-0203-44c6-8275-d20462ef5726_characterizeImage_946181_17.stderr
parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1174198c2d13206cb7412bdecf60bca6 with result from task 248
parsl.dataflow.dflow DEBUG: Task 248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/1723df11-f141-41e9-adef-c4f8b71c38cb_calibrate_946072_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/1723df11-f141-41e9-adef-c4f8b71c38cb_calibrate_946072_28.stderr
parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f27922de4804fb1f3e62b95f21bb9369 with result from task 1057
parsl.dataflow.dflow DEBUG: Task 1057 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash f46c81471dbe25929ac65820836555dc
parsl.dataflow.memoization INFO: Task 1058 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 977
parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac3e6519-3ffb-4b5f-8fba-970b90f67828_calibrate_946182_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac3e6519-3ffb-4b5f-8fba-970b90f67828_calibrate_946182_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/09e0ae8c-33c6-4d6a-ad9c-e5f3aaa044e5_characterizeImage_946182_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/09e0ae8c-33c6-4d6a-ad9c-e5f3aaa044e5_characterizeImage_946182_38.stderr
parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33fe6a6e462911ebe1cc2140b0b7c14e with result from task 559
parsl.dataflow.dflow DEBUG: Task 559 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 6b95c61fd9f825b3cde9703ad84b13ab
parsl.dataflow.memoization INFO: Task 560 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 978
parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/984607d3-1958-4812-931a-a6319165f53f_calibrate_946181_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/984607d3-1958-4812-931a-a6319165f53f_calibrate_946181_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/957d6b3e-00b4-4040-835e-bc8fd837c8ca_characterizeImage_946181_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/957d6b3e-00b4-4040-835e-bc8fd837c8ca_characterizeImage_946181_33.stderr
parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a712779b61cac41f48f6f61c46ebba8 with result from task 520
parsl.dataflow.dflow DEBUG: Task 520 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 7ac12ee75e6bbe45a40e33a4da2bbd05
parsl.dataflow.memoization INFO: Task 521 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 979
parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a72a750d-bf8b-46c4-a00b-b79480023c1c_calibrate_946073_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a72a750d-bf8b-46c4-a00b-b79480023c1c_calibrate_946073_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/42f58a95-05c1-4ce0-9db3-55ba00d47fe7_characterizeImage_946073_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/42f58a95-05c1-4ce0-9db3-55ba00d47fe7_characterizeImage_946073_24.stderr
parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d63a29217ce0354ea826ed11d3e401d with result from task 245
parsl.dataflow.dflow DEBUG: Task 245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/246598a3-1b2c-4e09-a8d7-c27b26d87261_calibrate_946072_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/246598a3-1b2c-4e09-a8d7-c27b26d87261_calibrate_946072_14.stderr
parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f827f6e6593b8aa095d18e61c74f21f with result from task 238
parsl.dataflow.dflow DEBUG: Task 238 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 239 has memoization hash b5408d7e16478444d56542dad4658117
parsl.dataflow.memoization INFO: Task 239 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 980
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/09036a4b-a5f6-45e6-9e18-c3b7c73be56d_calibrate_946181_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/09036a4b-a5f6-45e6-9e18-c3b7c73be56d_calibrate_946181_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e016afa8-63a3-4036-a393-075938c4f72d_characterizeImage_946181_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e016afa8-63a3-4036-a393-075938c4f72d_characterizeImage_946181_25.stderr
parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4aace4225735e9b47dc3f3a15396d24 with result from task 424
parsl.dataflow.dflow DEBUG: Task 424 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 425 has memoization hash dfbaecd2b4beff681f15c8089bbf570d
parsl.dataflow.memoization INFO: Task 425 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a40e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 981
parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7bf77f3d-4ac3-4109-80a8-c6df331d522a_calibrate_946182_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7bf77f3d-4ac3-4109-80a8-c6df331d522a_calibrate_946182_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/07e5f42d-acbf-4bda-801e-27ae5f9169d2_characterizeImage_946182_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/07e5f42d-acbf-4bda-801e-27ae5f9169d2_characterizeImage_946182_13.stderr
parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecd661160f83f40f94e0fcf9aa1271fb with result from task 538
parsl.dataflow.dflow DEBUG: Task 538 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 18913a8977c48db7f44cb93b61dbd44b
parsl.dataflow.memoization INFO: Task 539 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 982
parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cf26aaa2-80d2-454d-9f43-08c3c0550bfc_calibrate_946180_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cf26aaa2-80d2-454d-9f43-08c3c0550bfc_calibrate_946180_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c576b14d-7b11-47f3-9ce9-23c510c1d6d5_characterizeImage_946180_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c576b14d-7b11-47f3-9ce9-23c510c1d6d5_characterizeImage_946180_28.stderr
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ca457ac6aff66d7d9a3affeb4691d8a with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/0a3bf06b-3a70-476f-b292-c2b95db5716a_calibrate_946072_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/0a3bf06b-3a70-476f-b292-c2b95db5716a_calibrate_946072_13.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b3ce4f38c338f4f500b96dddeb1957ce with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/a1cb1b77-9700-467e-901d-72eeaa4b581c_calibrate_946072_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/a1cb1b77-9700-467e-901d-72eeaa4b581c_calibrate_946072_50.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 275
parsl.dataflow.strategy DEBUG: Executor multi has 275 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7060b1de6fa54ef049141d02156b124 with result from task 440
parsl.dataflow.dflow DEBUG: Task 440 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c2844cc6-72b2-4d10-98da-998d5c06c16e_calibrate_946182_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c2844cc6-72b2-4d10-98da-998d5c06c16e_calibrate_946182_17.stderr
parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f593f1cc0fc283882eb0ce9f4eac6e7c with result from task 934
parsl.dataflow.dflow DEBUG: Task 934 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 7b141df7609bac62d94df53be0c9c94d
parsl.dataflow.memoization INFO: Task 935 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 983
parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9c2912c9-1556-4601-87f3-ca7f33d97c57_calibrate_946182_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9c2912c9-1556-4601-87f3-ca7f33d97c57_calibrate_946182_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2c6740ef-083b-45d2-a9fb-6d1091b2c059_characterizeImage_946182_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2c6740ef-083b-45d2-a9fb-6d1091b2c059_characterizeImage_946182_3.stderr
parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3163036c5ba7d12779ffcc1396586351 with result from task 406
parsl.dataflow.dflow DEBUG: Task 406 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 407 has memoization hash f2a41aa67e7ab3f08ac2f6b0d5356a1e
parsl.dataflow.memoization INFO: Task 407 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a1fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 984
parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac6cccd0-56e2-4399-94d5-21c74775ff95_calibrate_946182_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac6cccd0-56e2-4399-94d5-21c74775ff95_calibrate_946182_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/921b14b8-b0c3-4a1f-8c01-f63bcd18af63_characterizeImage_946182_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/921b14b8-b0c3-4a1f-8c01-f63bcd18af63_characterizeImage_946182_48.stderr
parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cee1d03a99a3540097472dc19b5e47a9 with result from task 604
parsl.dataflow.dflow DEBUG: Task 604 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 7f0cb5f3e306b7c373c214a6fe6c5274
parsl.dataflow.memoization INFO: Task 605 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 985
parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/08b593b2-b065-4eea-b38f-9b501d971cd5_calibrate_946072_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/08b593b2-b065-4eea-b38f-9b501d971cd5_calibrate_946072_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c6e3b646-5a78-4d2e-ae07-28e044365f8c_characterizeImage_946072_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/c6e3b646-5a78-4d2e-ae07-28e044365f8c_characterizeImage_946072_23.stderr
parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95ef636019694ea625deb2ce2b91c906 with result from task 494
parsl.dataflow.dflow DEBUG: Task 494 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/79f13d75-620e-49ef-bde4-26dac762c8ed_calibrate_946074_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/79f13d75-620e-49ef-bde4-26dac762c8ed_calibrate_946074_51.stderr
parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20a2788639c2750e79817c710fc70463 with result from task 679
parsl.dataflow.dflow DEBUG: Task 679 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 4ac71e0027ea94e196d4a24be4a476a8
parsl.dataflow.memoization INFO: Task 680 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 986
parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/810c42b3-eed8-485e-bd47-d2ee86ef3365_calibrate_946072_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/810c42b3-eed8-485e-bd47-d2ee86ef3365_calibrate_946072_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9ff2e47f-e0b8-4ae9-b73f-88d7bd58a72c_characterizeImage_946072_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/9ff2e47f-e0b8-4ae9-b73f-88d7bd58a72c_characterizeImage_946072_4.stderr
parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e42776d9a33b75751653474d1c46365 with result from task 602
parsl.dataflow.dflow DEBUG: Task 602 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3a70abaa-3eef-46c7-8078-63d486b13b0d_calibrate_946180_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3a70abaa-3eef-46c7-8078-63d486b13b0d_calibrate_946180_10.stderr
parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8257ee9da52cbd619b293f0440dc2377 with result from task 929
parsl.dataflow.dflow DEBUG: Task 929 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/99c9a565-929e-45df-8245-c78aa0063fa5_calibrate_946074_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/99c9a565-929e-45df-8245-c78aa0063fa5_calibrate_946074_13.stderr
parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1ace59dc4d991644bbc7731f993ba1d with result from task 1087
parsl.dataflow.dflow DEBUG: Task 1087 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash a054d163be69b250850a29c60c18872f
parsl.dataflow.memoization INFO: Task 1088 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 987
parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9cb301c2-49df-4558-9ecb-8cc98167d31b_calibrate_946073_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9cb301c2-49df-4558-9ecb-8cc98167d31b_calibrate_946073_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ee13c141-2b31-450a-949b-5a4898c48c27_characterizeImage_946073_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/ee13c141-2b31-450a-949b-5a4898c48c27_characterizeImage_946073_12.stderr
parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0009f5052bed072ef4e59cb7e969eb6 with result from task 1045
parsl.dataflow.dflow DEBUG: Task 1045 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash f8ca3ac7bbf00e9471ed90f988df1855
parsl.dataflow.memoization INFO: Task 1046 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 988
parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/dda3d422-a709-4b1e-b0fb-52824a435cb3_calibrate_946181_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/dda3d422-a709-4b1e-b0fb-52824a435cb3_calibrate_946181_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/baa87add-d414-4e53-a350-38226fb50489_characterizeImage_946181_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/baa87add-d414-4e53-a350-38226fb50489_characterizeImage_946181_40.stderr
parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9996b983a28d981c965ade4a75b98265 with result from task 547
parsl.dataflow.dflow DEBUG: Task 547 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 548 has memoization hash a186830eaf21d187232f27d72216766b
parsl.dataflow.memoization INFO: Task 548 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 989
parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4de0082e-2162-4501-b2be-ed153b21f2c8_calibrate_946181_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4de0082e-2162-4501-b2be-ed153b21f2c8_calibrate_946181_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b104e42-bd90-4db0-8c74-351b45755143_characterizeImage_946181_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/2b104e42-bd90-4db0-8c74-351b45755143_characterizeImage_946181_38.stderr
parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64b5445ec17c4536f09463b5ee85edfd with result from task 304
parsl.dataflow.dflow DEBUG: Task 304 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 0f72db76c274d2bfd963e215d7e82f47
parsl.dataflow.memoization INFO: Task 305 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 990
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a872d42b-6e02-4baa-a215-94c9f9130163_calibrate_946182_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a872d42b-6e02-4baa-a215-94c9f9130163_calibrate_946182_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/688376e5-ca43-4984-bf39-f37c19ad65bd_characterizeImage_946182_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/688376e5-ca43-4984-bf39-f37c19ad65bd_characterizeImage_946182_4.stderr
parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 942c22038c4ba007e8147e373a92f08a with result from task 215
parsl.dataflow.dflow DEBUG: Task 215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/06d427da-f5ff-4404-b671-e806bf084464_calibrate_946180_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/06d427da-f5ff-4404-b671-e806bf084464_calibrate_946180_9.stderr
parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a3317fb8fea66e2f4b787f7e867c326b with result from task 725
parsl.dataflow.dflow DEBUG: Task 725 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/988a0797-0f7e-4236-a8cc-3099bd5a49ab_calibrate_946181_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/988a0797-0f7e-4236-a8cc-3099bd5a49ab_calibrate_946181_18.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7236e29880a18f7a32c964d347c11829 with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash ba4e79c10dcc501c286f3e9adc455145
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 991
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b1a0fd09-f8f8-44ac-9aa4-b26afde6fa09_calibrate_946074_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b1a0fd09-f8f8-44ac-9aa4-b26afde6fa09_calibrate_946074_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/94a3a18a-6cfc-493e-b8e6-42aba2e2dd4f_characterizeImage_946074_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/94a3a18a-6cfc-493e-b8e6-42aba2e2dd4f_characterizeImage_946074_7.stderr
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b51d9bb8e820282ab328eaa3f9f7a6a with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d22cc4f4-e759-4aab-8506-8231996f9049_calibrate_946181_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d22cc4f4-e759-4aab-8506-8231996f9049_calibrate_946181_55.stderr
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e26c6b37543ccafcace72db1e2a43b4 with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash e868da1746ca8db922b7ad5e97789c53
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 992
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/65c52be7-5b59-4aa9-99df-a98839f1b94f_calibrate_946180_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/65c52be7-5b59-4aa9-99df-a98839f1b94f_calibrate_946180_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/980d2f69-d781-4e6a-b1fe-9fc97a0693ba_characterizeImage_946180_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/980d2f69-d781-4e6a-b1fe-9fc97a0693ba_characterizeImage_946180_7.stderr
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c959292310298fc2641538df573712a4 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 0a42a3a93f3d5e3c14f3a16b25b9129a
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 993
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/74b3c647-b0fb-4323-afa2-933c658752a0_characterizeImage_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/74b3c647-b0fb-4323-afa2-933c658752a0_characterizeImage_946074_61.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fbbc5f6e6da7a71b72037ac9a24f269 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3cc4b7e-0c2f-4100-8775-6dd441dcc658_calibrate_946072_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3cc4b7e-0c2f-4100-8775-6dd441dcc658_calibrate_946072_60.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 267
parsl.dataflow.strategy DEBUG: Executor multi has 267 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e43ecd7d3062560e0ac59a72d7d92991 with result from task 482
parsl.dataflow.dflow DEBUG: Task 482 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/aa530f39-bfc4-42e8-99db-a9e61ee2d4a5_calibrate_946180_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/aa530f39-bfc4-42e8-99db-a9e61ee2d4a5_calibrate_946180_20.stderr
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a12b5760dd60055d6798cdce1c848df with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee2d2edf-3553-46f7-8abd-b5ea871c7289_calibrate_946180_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee2d2edf-3553-46f7-8abd-b5ea871c7289_calibrate_946180_62.stderr
parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2614104c5c8d29c3919a50f56bd6fc0 with result from task 308
parsl.dataflow.dflow DEBUG: Task 308 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6e97c7b-780b-4762-a19c-67dff364ec3d_calibrate_946074_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6e97c7b-780b-4762-a19c-67dff364ec3d_calibrate_946074_53.stderr
parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a630aba8cf44189263fe98c73e26366e with result from task 428
parsl.dataflow.dflow DEBUG: Task 428 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e723cfe2-ae88-4739-89f0-c7fab1aabc4f_calibrate_946074_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e723cfe2-ae88-4739-89f0-c7fab1aabc4f_calibrate_946074_45.stderr
parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d545f2ca3ced70050fd8afb9c32687ae with result from task 572
parsl.dataflow.dflow DEBUG: Task 572 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01d25c84-25e0-4e52-98d5-80d8f357719e_calibrate_946074_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01d25c84-25e0-4e52-98d5-80d8f357719e_calibrate_946074_16.stderr
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bed57f5360525db4ca16d5ab3467015c with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cc282278-09ea-40a6-a061-9857b55bdc26_calibrate_946073_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cc282278-09ea-40a6-a061-9857b55bdc26_calibrate_946073_11.stderr
parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 657200f68fd0117c8ab3bbef4a3097c7 with result from task 502
parsl.dataflow.dflow DEBUG: Task 502 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 56f93ed7b96ab459dfcd86270ca234a0
parsl.dataflow.memoization INFO: Task 503 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 994
parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3f73d4f-a89e-43ec-b7c3-079f4dcb71aa_calibrate_946181_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3f73d4f-a89e-43ec-b7c3-079f4dcb71aa_calibrate_946181_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e9c9115c-6283-4bbe-9edf-0129ba38c856_characterizeImage_946181_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e9c9115c-6283-4bbe-9edf-0129ba38c856_characterizeImage_946181_56.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b4ec23b99408ca8e6132d083245c599 with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34056ee1-e1a1-436d-9980-601a406ef2e8_calibrate_946181_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34056ee1-e1a1-436d-9980-601a406ef2e8_calibrate_946181_34.stderr
parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1933d0fd6881b92cc80c1df4d1b76134 with result from task 662
parsl.dataflow.dflow DEBUG: Task 662 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6f45e800-2dd6-4141-a402-5d8826cc6a00_calibrate_946073_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6f45e800-2dd6-4141-a402-5d8826cc6a00_calibrate_946073_8.stderr
parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3152b58db1da8f4e4f999b6851da97d3 with result from task 298
parsl.dataflow.dflow DEBUG: Task 298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 3c85991f667e92eff4e1cb30947d9bf7
parsl.dataflow.memoization INFO: Task 299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 995
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c10b0c38-2d8d-4d6e-8759-2bc12d8e4c3e_calibrate_946180_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c10b0c38-2d8d-4d6e-8759-2bc12d8e4c3e_calibrate_946180_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/64b03aa7-fb54-45cd-89c0-68f04970a9bf_characterizeImage_946180_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/64b03aa7-fb54-45cd-89c0-68f04970a9bf_characterizeImage_946180_52.stderr
parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 691b7287b497dd2c55af1ab2063b88ad with result from task 280
parsl.dataflow.dflow DEBUG: Task 280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 3dec3780dfd3112e45476978ce7ebcc3
parsl.dataflow.memoization INFO: Task 281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 996
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b84ef8ad-a546-45b6-b3a9-ed36c6fe8ce0_calibrate_946072_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b84ef8ad-a546-45b6-b3a9-ed36c6fe8ce0_calibrate_946072_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/de40cdd6-2497-4a56-83b9-ac194235c466_characterizeImage_946072_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/de40cdd6-2497-4a56-83b9-ac194235c466_characterizeImage_946072_16.stderr
parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81723455dca69cd3cb55d980634f1554 with result from task 1048
parsl.dataflow.dflow DEBUG: Task 1048 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 119aac5e083692d9aafd0511596f0e69
parsl.dataflow.memoization INFO: Task 1049 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 997
parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c1d21ff8-9e82-4abe-849d-ad4466655add_calibrate_946181_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c1d21ff8-9e82-4abe-849d-ad4466655add_calibrate_946181_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0cf01beb-7555-40a8-88ba-e574eae99326_characterizeImage_946181_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/0cf01beb-7555-40a8-88ba-e574eae99326_characterizeImage_946181_11.stderr
parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1dc3dd72310ffd1ca6fe9d8b1472291c with result from task 695
parsl.dataflow.dflow DEBUG: Task 695 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/aa19bef3-5e40-4626-a662-640cfd97b65b_calibrate_946072_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/aa19bef3-5e40-4626-a662-640cfd97b65b_calibrate_946072_9.stderr
parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5de0e558736fecf308316c02222ee11d with result from task 673
parsl.dataflow.dflow DEBUG: Task 673 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 2fbcdc366777ddcff62f8715d526f700
parsl.dataflow.memoization INFO: Task 674 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 998
parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8bebe307-0614-47d1-9cf7-5b7d0e06833d_calibrate_946072_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8bebe307-0614-47d1-9cf7-5b7d0e06833d_calibrate_946072_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a38f5c90-94f0-494b-b2cb-4b63cf74fb89_characterizeImage_946072_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/a38f5c90-94f0-494b-b2cb-4b63cf74fb89_characterizeImage_946072_43.stderr
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95de1123a20efca3aa9f5c68b5ab5beb with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash c38f0ee1b8d0871e6a64fe06c84804cc
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 999
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f40be76b-a8ea-4dd1-8532-875952591beb_calibrate_946182_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f40be76b-a8ea-4dd1-8532-875952591beb_calibrate_946182_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a2df69b9-bec6-4c89-96f9-37203b9580ac_characterizeImage_946182_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a2df69b9-bec6-4c89-96f9-37203b9580ac_characterizeImage_946182_51.stderr
parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2294204327c36c28a35b542e788e44e1 with result from task 479
parsl.dataflow.dflow DEBUG: Task 479 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3734002-702a-402d-9414-ef474961bc09_calibrate_946072_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/c3734002-702a-402d-9414-ef474961bc09_calibrate_946072_48.stderr
parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18871936fccc40b3c723b270eaaa3c20 with result from task 842
parsl.dataflow.dflow DEBUG: Task 842 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fc8da50f-6a55-4f1b-83ef-8216e83dbeb6_calibrate_946074_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fc8da50f-6a55-4f1b-83ef-8216e83dbeb6_calibrate_946074_47.stderr
parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 900d97c4d808b0a1a67857ad04d8f873 with result from task 955
parsl.dataflow.dflow DEBUG: Task 955 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 4282fe70ed1118f3688a45c16e406a9e
parsl.dataflow.memoization INFO: Task 956 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 1000
parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/b1626dd8-fa6f-42d8-b4a9-6b1023371793_calibrate_946181_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/b1626dd8-fa6f-42d8-b4a9-6b1023371793_calibrate_946181_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b40a7168-3924-484e-b771-beadf85db982_characterizeImage_946181_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b40a7168-3924-484e-b771-beadf85db982_characterizeImage_946181_54.stderr
parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20d4be282ee7da5c917250a26128336f with result from task 359
parsl.dataflow.dflow DEBUG: Task 359 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/75b8a4af-f988-4365-83f2-c8ab138c0fa3_calibrate_946072_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/75b8a4af-f988-4365-83f2-c8ab138c0fa3_calibrate_946072_38.stderr
parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd0e2a4c51bab36eddff665f2ab68076 with result from task 1066
parsl.dataflow.dflow DEBUG: Task 1066 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 633ab14526aa68db954331bc700a6523
parsl.dataflow.memoization INFO: Task 1067 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 1001
parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/2cfbe8b0-5360-470b-a3d8-6b907d6fb9f7_calibrate_946180_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/2cfbe8b0-5360-470b-a3d8-6b907d6fb9f7_calibrate_946180_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3da21804-299e-416c-926e-4e07eab48d20_characterizeImage_946180_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/3da21804-299e-416c-926e-4e07eab48d20_characterizeImage_946180_57.stderr
parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2497f6a3b5970c10225d0a2ec1b38586 with result from task 676
parsl.dataflow.dflow DEBUG: Task 676 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 5dcb01ff083da018fc35c5d5d1a7b4ab
parsl.dataflow.memoization INFO: Task 677 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308935580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 1002
parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8d9c4102-cedb-4b56-9989-7a32e0022c44_calibrate_946074_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8d9c4102-cedb-4b56-9989-7a32e0022c44_calibrate_946074_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e8d73969-9155-4930-a0b1-b8b2745e2933_characterizeImage_946074_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e8d73969-9155-4930-a0b1-b8b2745e2933_characterizeImage_946074_1.stderr
2024-03-13 21:38:41 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c4a5c73af2d84bb70aa3f82bc96fb22 with result from task 875
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow DEBUG: Task 875 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f6472e3f-bbc4-4653-9f89-56c5ee132616_calibrate_946181_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f6472e3f-bbc4-4653-9f89-56c5ee132616_calibrate_946181_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:42 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8ac2f50331004d3d2fa2725e1b24e2a with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash b798c8d07528299543fd6240baa8bc20
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 1003
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3adfdaa3-3aeb-496c-a1c6-40d44bba918c_calibrate_946073_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3adfdaa3-3aeb-496c-a1c6-40d44bba918c_calibrate_946073_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a6c92cb1-d95e-4539-9b7c-fed22924c4e0_characterizeImage_946073_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/a6c92cb1-d95e-4539-9b7c-fed22924c4e0_characterizeImage_946073_56.stderr
parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b365617e81f210ff1c33ddea1495323 with result from task 458
parsl.dataflow.dflow DEBUG: Task 458 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ce9877f1-0e3b-4a48-97a9-857a92ae289f_calibrate_946180_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ce9877f1-0e3b-4a48-97a9-857a92ae289f_calibrate_946180_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a374bdcc92d9792a30c1b413ec4543f with result from task 671
parsl.dataflow.dflow DEBUG: Task 671 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/29e96ea3-5939-4901-a8c1-ed9383837feb_calibrate_946182_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/29e96ea3-5939-4901-a8c1-ed9383837feb_calibrate_946182_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:43 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8dbc2a3729af60c4c50b91571b3f9193 with result from task 974
parsl.dataflow.dflow DEBUG: Task 974 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f7e633ed-d846-4dc0-a2c8-47210019f932_calibrate_946072_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f7e633ed-d846-4dc0-a2c8-47210019f932_calibrate_946072_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e94d090b0a7a92bf5a1b0a5beec8a00 with result from task 569
parsl.dataflow.dflow DEBUG: Task 569 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8f0f71f8-674e-4125-af82-6e9677b41d94_calibrate_946072_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8f0f71f8-674e-4125-af82-6e9677b41d94_calibrate_946072_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d10a86750913ac6a97d103954133a0a1 with result from task 638
parsl.dataflow.dflow DEBUG: Task 638 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/489dc325-fd6c-4139-8733-9ff230e16d70_calibrate_946073_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/489dc325-fd6c-4139-8733-9ff230e16d70_calibrate_946073_32.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60439821f9fee119861b70255522ed8a with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/aa8edcc2-dbaa-46bc-9bb4-562ab6cdd7aa_calibrate_946181_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/aa8edcc2-dbaa-46bc-9bb4-562ab6cdd7aa_calibrate_946181_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fcc126503d05f990a788f409c3032855 with result from task 505
parsl.dataflow.dflow DEBUG: Task 505 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 75320902fe218fcec539d567b2c152b4
parsl.dataflow.memoization INFO: Task 506 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 1004
parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5d471be8-0189-4b12-a638-86f5b63b806a_calibrate_946181_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5d471be8-0189-4b12-a638-86f5b63b806a_calibrate_946181_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7965a59-990d-46ff-a84d-e5f71644da48_characterizeImage_946181_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d7965a59-990d-46ff-a84d-e5f71644da48_characterizeImage_946181_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:44 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee3a947fcb5b60be944d22a125d20d59 with result from task 286
parsl.dataflow.dflow DEBUG: Task 286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 287 has memoization hash f148c1a7ef17377522a55f7e86a76629
parsl.dataflow.memoization INFO: Task 287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 1005
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/503cac68-f898-409d-b803-48b9ea3fc2f9_calibrate_946072_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/503cac68-f898-409d-b803-48b9ea3fc2f9_calibrate_946072_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/419feebc-a639-41c6-a913-a7630e35ff13_characterizeImage_946072_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/419feebc-a639-41c6-a913-a7630e35ff13_characterizeImage_946072_29.stderr
parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9767f240982f26b1641dcb307bcb58ad with result from task 892
parsl.dataflow.dflow DEBUG: Task 892 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 893 has memoization hash bddbe7dac7309eaa7ed32a63144236e6
parsl.dataflow.memoization INFO: Task 893 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 1006
parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7286120-fdbd-4a9d-8157-50602ba3e535_calibrate_946073_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7286120-fdbd-4a9d-8157-50602ba3e535_calibrate_946073_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/eb303985-2bc0-4788-960c-f872b6187bc3_characterizeImage_946073_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/eb303985-2bc0-4788-960c-f872b6187bc3_characterizeImage_946073_52.stderr
parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6dad027af2becbd0ae157491bb7f960f with result from task 704
parsl.dataflow.dflow DEBUG: Task 704 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9a568c54-ea78-4922-8518-314387c3c656_calibrate_946182_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9a568c54-ea78-4922-8518-314387c3c656_calibrate_946182_57.stderr
parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc25420bdeb558916cfd7ecfccdc8795 with result from task 895
parsl.dataflow.dflow DEBUG: Task 895 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 2cb43ef040e20bf9c872aa6c0a72654b
parsl.dataflow.memoization INFO: Task 896 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 1007
parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3909df4e-a3e3-401b-b36b-fa8e04b4d47a_calibrate_946072_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3909df4e-a3e3-401b-b36b-fa8e04b4d47a_calibrate_946072_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/11274884-ee40-4aa9-9475-f871aaf39c56_characterizeImage_946072_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/11274884-ee40-4aa9-9475-f871aaf39c56_characterizeImage_946072_31.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73d86d12251351bd6b7986306767dd29 with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/afaa6fae-c053-41ff-a19a-b3bee089c99e_calibrate_946072_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/afaa6fae-c053-41ff-a19a-b3bee089c99e_calibrate_946072_19.stderr
parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23b1ccbda8c45f600b00cf469cccc246 with result from task 913
parsl.dataflow.dflow DEBUG: Task 913 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 914 has memoization hash f85e7e08d6dc75f47373d4a4f2788316
parsl.dataflow.memoization INFO: Task 914 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 1008
parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7cf62449-4fc8-4468-b6d4-294beb041157_calibrate_946073_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7cf62449-4fc8-4468-b6d4-294beb041157_calibrate_946073_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/bf812fdb-66b9-44b9-b876-16bb6c81f2dc_characterizeImage_946073_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/bf812fdb-66b9-44b9-b876-16bb6c81f2dc_characterizeImage_946073_35.stderr
parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 799cb18df949caab270841927a00ae25 with result from task 448
parsl.dataflow.dflow DEBUG: Task 448 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 44d251afd58e584f8b5e30677baf2048
parsl.dataflow.memoization INFO: Task 449 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a42700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 1009
parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/845c96a6-90ca-464e-933f-530e405b82d1_calibrate_946072_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/845c96a6-90ca-464e-933f-530e405b82d1_calibrate_946072_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4deeba74-f8d5-4f2e-b395-e285b5ec1950_characterizeImage_946072_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/4deeba74-f8d5-4f2e-b395-e285b5ec1950_characterizeImage_946072_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 373dc6c6b9b68b99a452af0252ece118 with result from task 1090
parsl.dataflow.dflow DEBUG: Task 1090 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash e30ae19cbe5503d8468c5439a60f0553
parsl.dataflow.memoization INFO: Task 1091 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308734680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1010
parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/657b5de9-a944-4366-b4e2-60b145f2cf9d_calibrate_946181_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/657b5de9-a944-4366-b4e2-60b145f2cf9d_calibrate_946181_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/29221276-e60a-4d2e-a3a2-251e9e3d5458_characterizeImage_946181_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/29221276-e60a-4d2e-a3a2-251e9e3d5458_characterizeImage_946181_8.stderr
parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2338b807748c57ad97898eec2ae35522 with result from task 976
parsl.dataflow.dflow DEBUG: Task 976 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 977 has memoization hash ddc11f2f92104b1ce5f04c6375a675eb
parsl.dataflow.memoization INFO: Task 977 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 1011
parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/447bd682-2688-4a20-bdcf-7b6bb37c7d99_calibrate_946180_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/447bd682-2688-4a20-bdcf-7b6bb37c7d99_calibrate_946180_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8b6cc605-e486-4428-976f-0b656fb3f980_characterizeImage_946180_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8b6cc605-e486-4428-976f-0b656fb3f980_characterizeImage_946180_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b88cbb80e76d21b66ac1c414703b0c61 with result from task 496
parsl.dataflow.dflow DEBUG: Task 496 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 6fe6f2b1af4e54cb60b327d2359a0dd8
parsl.dataflow.memoization INFO: Task 497 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a75940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1012
parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c4efff05-24d5-45fb-8d78-50ea0dddf600_calibrate_946180_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c4efff05-24d5-45fb-8d78-50ea0dddf600_calibrate_946180_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8d735523-99bf-4204-b6e7-d19a1c18ecf5_characterizeImage_946180_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/8d735523-99bf-4204-b6e7-d19a1c18ecf5_characterizeImage_946180_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53f29d3e714443e699036c87c1ccf242 with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 9e6d3814252f6cc7ab351f2b08420868
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 1013
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc2a16b0-0d59-4385-82ba-fc7bb8c19b82_calibrate_946180_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc2a16b0-0d59-4385-82ba-fc7bb8c19b82_calibrate_946180_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/59a768e0-5ec4-4776-ac99-be9d661e8073_characterizeImage_946180_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/59a768e0-5ec4-4776-ac99-be9d661e8073_characterizeImage_946180_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4be02e23634d89653b1f2c42a7e5b18 with result from task 613
parsl.dataflow.dflow DEBUG: Task 613 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 455f837289a8fb9f102d56178d913411
parsl.dataflow.memoization INFO: Task 614 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 1014
parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c3c52260-7935-47d0-bb09-a7bc02ee4f78_calibrate_946182_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c3c52260-7935-47d0-bb09-a7bc02ee4f78_calibrate_946182_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/788caf51-6ecd-4b02-a70d-744dd2624761_characterizeImage_946182_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/788caf51-6ecd-4b02-a70d-744dd2624761_characterizeImage_946182_34.stderr
parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 194b62e565289953f18e8049b3a62e9d with result from task 902
parsl.dataflow.dflow DEBUG: Task 902 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f6bdc437-e632-43c0-8401-cdbd02d604e4_calibrate_946072_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f6bdc437-e632-43c0-8401-cdbd02d604e4_calibrate_946072_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:45 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry efe4e8018780b7c926a76b62b9c1fffb with result from task 193
parsl.dataflow.dflow DEBUG: Task 193 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 194 has memoization hash d7f51a02c091ea5f5db832551655ad92
parsl.dataflow.memoization INFO: Task 194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 1015
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/131a090a-2d98-4a2c-9a6b-b9b0fb6ee254_calibrate_946182_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/131a090a-2d98-4a2c-9a6b-b9b0fb6ee254_calibrate_946182_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0e2ce6c5-7bd4-46ab-8791-41e4bf8e813a_characterizeImage_946182_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0e2ce6c5-7bd4-46ab-8791-41e4bf8e813a_characterizeImage_946182_23.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f030bc6dd091b1be8b54e9cbae44b5bc with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash ba83095a96e53a77aae3d322c61bac43
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 1016
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9afa961a-aec1-4c67-a0fa-c45ee48f6c0e_calibrate_946182_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9afa961a-aec1-4c67-a0fa-c45ee48f6c0e_calibrate_946182_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2e686eb1-55f2-473b-9aeb-12b777c57b7d_characterizeImage_946182_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2e686eb1-55f2-473b-9aeb-12b777c57b7d_characterizeImage_946182_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b01af19a1f455f6f4f2e90e913e1f6e6 with result from task 971
parsl.dataflow.dflow DEBUG: Task 971 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/987ef3c6-326a-4b80-8f5c-6fd3073a56ce_calibrate_946072_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/987ef3c6-326a-4b80-8f5c-6fd3073a56ce_calibrate_946072_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:46 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42b21c639cca561a201179450367845f with result from task 658
parsl.dataflow.dflow DEBUG: Task 658 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 659 has memoization hash d0573a2325676be80015ab8c9d25fcb0
parsl.dataflow.memoization INFO: Task 659 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 1017
parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3957b707-722c-4a58-b0ae-1e29025b4731_calibrate_946180_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3957b707-722c-4a58-b0ae-1e29025b4731_calibrate_946180_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d44b3d8b-263f-474c-a366-d3f0328e0530_characterizeImage_946180_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d44b3d8b-263f-474c-a366-d3f0328e0530_characterizeImage_946180_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec35b846ff3c4ab7d63e8b45265fb875 with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/09006943-70cd-4b0b-bfd2-e826cb75286b_calibrate_946180_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/09006943-70cd-4b0b-bfd2-e826cb75286b_calibrate_946180_19.stderr
parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 600a7870e18e7b1616a82bb00622fc19 with result from task 1081
parsl.dataflow.dflow DEBUG: Task 1081 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash d253cf418af3cd7cbc3a4f53ee9538ab
parsl.dataflow.memoization INFO: Task 1082 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086ebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1018
parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e30248a5-135e-46fd-bf87-8153bc95a7b3_calibrate_946182_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e30248a5-135e-46fd-bf87-8153bc95a7b3_calibrate_946182_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6580c083-7cde-4e89-ae2c-052fc200a461_characterizeImage_946182_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6580c083-7cde-4e89-ae2c-052fc200a461_characterizeImage_946182_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 049cc23340ad0768c944eb2c87b81084 with result from task 296
parsl.dataflow.dflow DEBUG: Task 296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/be0735f0-c97c-4473-a404-f18078e3b80d_calibrate_946072_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/be0735f0-c97c-4473-a404-f18078e3b80d_calibrate_946072_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:47 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14c6c3072a559e399bc8bdd6f227c3f5 with result from task 368
parsl.dataflow.dflow DEBUG: Task 368 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/3d52e8ef-a22c-4215-9d53-2ca44a0aaf95_calibrate_946074_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/3d52e8ef-a22c-4215-9d53-2ca44a0aaf95_calibrate_946074_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 29c0cbb4fbde79df4832238c7a67763d with result from task 455
parsl.dataflow.dflow DEBUG: Task 455 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/49841fcd-5076-4f1f-bdeb-bb51f60ea2ef_calibrate_946181_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/49841fcd-5076-4f1f-bdeb-bb51f60ea2ef_calibrate_946181_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 233e963b5af5dc7fff47eee3905ea9f4 with result from task 289
parsl.dataflow.dflow DEBUG: Task 289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 44996f025bfcd4db35bb38b54addfc18
parsl.dataflow.memoization INFO: Task 290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 1019
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ebb1f066-f472-4f07-8e70-ba0cfa3fda34_calibrate_946180_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ebb1f066-f472-4f07-8e70-ba0cfa3fda34_calibrate_946180_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/04024df1-8589-4c48-8dcf-6f1c1e951a59_characterizeImage_946180_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/04024df1-8589-4c48-8dcf-6f1c1e951a59_characterizeImage_946180_44.stderr
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8a250875b313b1f8b0540938dad4d93 with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash ccf961a191094fbf2b090b8797fbc63c
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 1020
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b42e89b2-0fa4-46d2-a85f-4767ee26c796_calibrate_946180_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b42e89b2-0fa4-46d2-a85f-4767ee26c796_calibrate_946180_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5fc3035b-f3ed-41e2-be7a-61526e418679_characterizeImage_946180_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/5fc3035b-f3ed-41e2-be7a-61526e418679_characterizeImage_946180_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:48 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cbf884eae09f1a213841dffe230a7e00 with result from task 578
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow DEBUG: Task 578 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/897799e8-82cb-41dc-bccc-c132e3128dcc_calibrate_946180_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/897799e8-82cb-41dc-bccc-c132e3128dcc_calibrate_946180_27.stderr
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7933cc12381c16e0418f7e83be65c46b with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c4db8bc4-387f-416e-81c5-1fa0c4ad1c93_calibrate_946074_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c4db8bc4-387f-416e-81c5-1fa0c4ad1c93_calibrate_946074_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9211f70f3b19b9a8cf0a77a183556d85 with result from task 401
parsl.dataflow.dflow DEBUG: Task 401 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/79aaef45-2b2e-45d9-85da-393ac89f2a0f_calibrate_946181_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/79aaef45-2b2e-45d9-85da-393ac89f2a0f_calibrate_946181_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:49 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5a52d5f8cb0c7ffceed2f16cdbb33b8 with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 43bff9c95506d16b52b2d50d74c14ffd
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f8a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 1021
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/3d0fb75f-5dbc-4149-b0db-e6e6939b71d8_calibrate_946181_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/3d0fb75f-5dbc-4149-b0db-e6e6939b71d8_calibrate_946181_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50ef5a04-6dab-4c5c-9085-b0e09b0aef6a_characterizeImage_946181_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/50ef5a04-6dab-4c5c-9085-b0e09b0aef6a_characterizeImage_946181_36.stderr
parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8b4130c5e99dbf627fc3fa2f21bf890 with result from task 347
parsl.dataflow.dflow DEBUG: Task 347 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/57185630-5a52-47d8-9bbc-5ec56e9bb686_calibrate_946074_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/57185630-5a52-47d8-9bbc-5ec56e9bb686_calibrate_946074_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfc1a5746149deaba6291130e6af561e with result from task 257
parsl.dataflow.dflow DEBUG: Task 257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/432484a1-14a3-433e-a20c-ddc0756f0e9e_calibrate_946180_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/432484a1-14a3-433e-a20c-ddc0756f0e9e_calibrate_946180_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68ba42d7c7d1873deb7ead96f130b702 with result from task 826
parsl.dataflow.dflow DEBUG: Task 826 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 827 has memoization hash ce9d1a0ff63c2913dd979b87f1f65efa
parsl.dataflow.memoization INFO: Task 827 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 1022
parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/cda2399e-80bc-4bb2-8c7c-389f04ee00e2_calibrate_946181_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/cda2399e-80bc-4bb2-8c7c-389f04ee00e2_calibrate_946181_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5c2fcdb3-d753-4e0b-8b97-2afa6b0e86a3_characterizeImage_946181_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/5c2fcdb3-d753-4e0b-8b97-2afa6b0e86a3_characterizeImage_946181_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:50 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0dee297654017dccfbe8656c40eea94 with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d212850d-3c9c-4710-b719-be7a570a86d7_calibrate_946074_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d212850d-3c9c-4710-b719-be7a570a86d7_calibrate_946074_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:51 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78518e85415479c553d00c885eeb22b1 with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 0647c9d90bfee012e4e6c922265d4aa1
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 1023
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34dda894-cc51-4efa-ae11-8f6a884bd5fb_calibrate_946181_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34dda894-cc51-4efa-ae11-8f6a884bd5fb_calibrate_946181_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/f73b3b4e-83f7-417b-99cc-2dafd01bb074_characterizeImage_946181_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/f73b3b4e-83f7-417b-99cc-2dafd01bb074_characterizeImage_946181_61.stderr
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f176528d8b44913c58fb4af766267ed with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c36ef7b7-0bbd-437d-a0d2-02e771d53d98_calibrate_946074_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c36ef7b7-0bbd-437d-a0d2-02e771d53d98_calibrate_946074_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4630bfecad5a20df73a1d5204f6562d5 with result from task 728
parsl.dataflow.dflow DEBUG: Task 728 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/423716c4-6e40-4454-b911-21cb7abc175c_calibrate_946180_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/423716c4-6e40-4454-b911-21cb7abc175c_calibrate_946180_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef440dd7c449b04977e82e079ea770c5 with result from task 937
parsl.dataflow.dflow DEBUG: Task 937 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 4921b434cb89855a8f4f777ce25bb21a
parsl.dataflow.memoization INFO: Task 938 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1024
parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f39d3afe-a6a3-4c3a-9169-e7912bee80ed_calibrate_946181_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f39d3afe-a6a3-4c3a-9169-e7912bee80ed_calibrate_946181_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ecec1eec-a978-4110-8d36-89d4ac8bf07d_characterizeImage_946181_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/ecec1eec-a978-4110-8d36-89d4ac8bf07d_characterizeImage_946181_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:52 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7fdfe496656d26727ca962ac386b3df with result from task 1052
parsl.dataflow.dflow DEBUG: Task 1052 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c03d0365-96f5-4410-9508-a6a107468bc0_calibrate_946181_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c03d0365-96f5-4410-9508-a6a107468bc0_calibrate_946181_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:53 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:53 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:53 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:53 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67b33bfe3f2b0819053d0c6cb8cb414a with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 65300f49175b144aae04909f7a6ef247
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 1025
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/32593b3b-dc54-4250-b016-97b880c5d419_calibrate_946182_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/32593b3b-dc54-4250-b016-97b880c5d419_calibrate_946182_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/41eabc99-0b4d-4672-8589-f6df5c705122_characterizeImage_946182_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/41eabc99-0b4d-4672-8589-f6df5c705122_characterizeImage_946182_44.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a61dab9813e6b5cd16359090bd2dd131 with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 51f122101375fbf047f426a627a9e93a
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 1026
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ea6708c5-088f-4021-aaa0-67e04c80d352_calibrate_946182_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ea6708c5-088f-4021-aaa0-67e04c80d352_calibrate_946182_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bb24e41a-813e-46f1-a466-cc104fb0a26f_characterizeImage_946182_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bb24e41a-813e-46f1-a466-cc104fb0a26f_characterizeImage_946182_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:53 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:38:53 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 231
parsl.dataflow.strategy DEBUG: Executor multi has 231 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 231
parsl.dataflow.strategy DEBUG: Executor multi has 231 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30f9e8b362f71480027c428386361d06 with result from task 523
parsl.dataflow.dflow DEBUG: Task 523 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 524 has memoization hash b1fde992dc2c59fbf2d48d5f78704307
parsl.dataflow.memoization INFO: Task 524 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a77560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 1027
parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/41518cc6-5c6d-4b71-ad17-a2ee250df8a3_calibrate_946181_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/41518cc6-5c6d-4b71-ad17-a2ee250df8a3_calibrate_946181_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/aad2c7b1-b8ed-42df-9daa-6ad4ac833ce5_characterizeImage_946181_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/aad2c7b1-b8ed-42df-9daa-6ad4ac833ce5_characterizeImage_946181_43.stderr
parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76b559113486a13aa239911c3ee57c2c with result from task 607
parsl.dataflow.dflow DEBUG: Task 607 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 32ec5e51cdb5e2f8615ae731357f3bad
parsl.dataflow.memoization INFO: Task 608 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 1028
parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3f5195f9-0153-4eba-bb39-660310b03ff1_calibrate_946182_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3f5195f9-0153-4eba-bb39-660310b03ff1_calibrate_946182_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/64eb9a45-bc10-43d1-95b1-2a371f4a25a0_characterizeImage_946182_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/64eb9a45-bc10-43d1-95b1-2a371f4a25a0_characterizeImage_946182_45.stderr
parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b959283bda1e1980391f7e41152e5dd2 with result from task 515
parsl.dataflow.dflow DEBUG: Task 515 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d9c2e574-ef21-478f-a836-fb3921cbb956_calibrate_946072_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/d9c2e574-ef21-478f-a836-fb3921cbb956_calibrate_946072_46.stderr
parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5ca859bc05eb03690031fad588e1afc with result from task 464
parsl.dataflow.dflow DEBUG: Task 464 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3762998-7a2d-42bc-bae5-6f0f687563c3_calibrate_946181_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3762998-7a2d-42bc-bae5-6f0f687563c3_calibrate_946181_57.stderr
parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 100b600d4983fb3dab12841b0020aed7 with result from task 626
parsl.dataflow.dflow DEBUG: Task 626 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6cd5551e-82b4-4ac2-93dd-833ff5bf11ce_calibrate_946072_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/6cd5551e-82b4-4ac2-93dd-833ff5bf11ce_calibrate_946072_36.stderr
parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b68298a38d1fe94c24a45cbe0c169ae with result from task 686
parsl.dataflow.dflow DEBUG: Task 686 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8d7cceae-0e24-4c0e-b812-991354e3b417_calibrate_946072_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8d7cceae-0e24-4c0e-b812-991354e3b417_calibrate_946072_27.stderr
parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12e1d09d3661440932f42de7174901d2 with result from task 395
parsl.dataflow.dflow DEBUG: Task 395 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c5f73a17-d7b8-45c2-94ca-a9c8db12c12a_calibrate_946181_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c5f73a17-d7b8-45c2-94ca-a9c8db12c12a_calibrate_946181_37.stderr
parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 431d0bc89efa28ecd82d26624b1e086a with result from task 812
parsl.dataflow.dflow DEBUG: Task 812 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18d4836d-f5e8-4f51-86e3-cb4b54e3d68c_calibrate_946180_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18d4836d-f5e8-4f51-86e3-cb4b54e3d68c_calibrate_946180_15.stderr
parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74a3318ded735d2e6f6ccb7a77c05942 with result from task 242
parsl.dataflow.dflow DEBUG: Task 242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8255522f-081f-4a1f-8a98-8dec35dbffa5_calibrate_946182_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8255522f-081f-4a1f-8a98-8dec35dbffa5_calibrate_946182_11.stderr
parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b040e8a583310d132054acb7e658915f with result from task 452
parsl.dataflow.dflow DEBUG: Task 452 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/060b33d4-1a74-424f-b03c-2faa4e91ee4b_calibrate_946181_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/060b33d4-1a74-424f-b03c-2faa4e91ee4b_calibrate_946181_53.stderr
parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dac2f60e20db33d7e3c1287b64c865d6 with result from task 259
parsl.dataflow.dflow DEBUG: Task 259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 260 has memoization hash c5f40093c793f2a9f711795465de1211
parsl.dataflow.memoization INFO: Task 260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 1029
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b5e72b97-8a2e-4a66-8523-c510dae0638f_calibrate_946182_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b5e72b97-8a2e-4a66-8523-c510dae0638f_calibrate_946182_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/810b7c05-e801-436e-8015-d4e499d71ecc_characterizeImage_946182_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/810b7c05-e801-436e-8015-d4e499d71ecc_characterizeImage_946182_30.stderr
parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed359de278313e568e615113795826f0 with result from task 908
parsl.dataflow.dflow DEBUG: Task 908 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0a31e6ca-48fc-4598-9464-cd3bba240d1f_calibrate_946182_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0a31e6ca-48fc-4598-9464-cd3bba240d1f_calibrate_946182_46.stderr
parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da939469ad61317cad5e2e70e31994b9 with result from task 655
parsl.dataflow.dflow DEBUG: Task 655 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 291f09a0604de839c5455dc73a5669e9
parsl.dataflow.memoization INFO: Task 656 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088efba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 1030
parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8599c38c-3a5b-4924-85d8-09e5434104b4_calibrate_946180_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8599c38c-3a5b-4924-85d8-09e5434104b4_calibrate_946180_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e6f803a0-0148-4275-91c3-36a3261d3d75_characterizeImage_946180_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e6f803a0-0148-4275-91c3-36a3261d3d75_characterizeImage_946180_21.stderr
parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 588f307652ad061e6ae3a71bdf52ab3c with result from task 314
parsl.dataflow.dflow DEBUG: Task 314 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0a2f8478-5a76-4af5-92ef-428cc7bf5441_calibrate_946074_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0a2f8478-5a76-4af5-92ef-428cc7bf5441_calibrate_946074_11.stderr
parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32164ea6a01a1072e09a4a631df9c06b with result from task 485
parsl.dataflow.dflow DEBUG: Task 485 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/99772fea-2a4b-4d07-8df8-03098eb76fb6_calibrate_946073_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/99772fea-2a4b-4d07-8df8-03098eb76fb6_calibrate_946073_27.stderr
parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e1e5d5c0398d221713eb6b3ac08d4b99 with result from task 683
parsl.dataflow.dflow DEBUG: Task 683 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/66d99a80-20f2-4bee-9183-bb154ea58320_calibrate_946072_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/66d99a80-20f2-4bee-9183-bb154ea58320_calibrate_946072_22.stderr
parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ef77c69ce350879bbce7d4770c83d9f with result from task 584
parsl.dataflow.dflow DEBUG: Task 584 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/6853380c-03f4-4c8d-b860-0caf2a17aa10_calibrate_946182_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/6853380c-03f4-4c8d-b860-0caf2a17aa10_calibrate_946182_33.stderr
parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43000d8fc74a289be31a77d09382346b with result from task 592
parsl.dataflow.dflow DEBUG: Task 592 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 593 has memoization hash ee24ef9360153ec94dc36caa80a652f4
parsl.dataflow.memoization INFO: Task 593 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 1031
parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/64e77cba-6c0f-4797-bee6-c46834052cc1_calibrate_946181_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/64e77cba-6c0f-4797-bee6-c46834052cc1_calibrate_946181_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b7bb503d-b30f-4994-a3b4-231ff7bf538b_characterizeImage_946181_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/b7bb503d-b30f-4994-a3b4-231ff7bf538b_characterizeImage_946181_10.stderr
parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 269baf549f10c80de71493eadc475bbe with result from task 374
parsl.dataflow.dflow DEBUG: Task 374 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/d25d88ab-f9e5-4d31-a1c1-5734af17c324_calibrate_946182_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/d25d88ab-f9e5-4d31-a1c1-5734af17c324_calibrate_946182_24.stderr
parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b971996ca70c35d7438b8a1a2d9f4dd with result from task 278
parsl.dataflow.dflow DEBUG: Task 278 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aed8ca06-1de3-4b27-988d-3b48aa2a3393_calibrate_946073_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/aed8ca06-1de3-4b27-988d-3b48aa2a3393_calibrate_946073_49.stderr
parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4f694d6066b478303e84956dbcc10e9 with result from task 530
parsl.dataflow.dflow DEBUG: Task 530 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/53682e6c-22a9-4cf9-9710-7ff987536b46_calibrate_946073_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/53682e6c-22a9-4cf9-9710-7ff987536b46_calibrate_946073_34.stderr
parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98f7f2936b7bd473484a0f946d8d5422 with result from task 949
parsl.dataflow.dflow DEBUG: Task 949 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 3dd739d83abe77fa2726692542e3ab81
parsl.dataflow.memoization INFO: Task 950 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 1032
parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/da8225bf-1f1a-45ed-84bb-a2a4a9efd4e8_calibrate_946181_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/da8225bf-1f1a-45ed-84bb-a2a4a9efd4e8_calibrate_946181_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a24bad96-f3a1-4200-92ca-def46aa70038_characterizeImage_946181_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/a24bad96-f3a1-4200-92ca-def46aa70038_characterizeImage_946181_24.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 214
parsl.dataflow.strategy DEBUG: Executor multi has 214 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb817ddd3e6d12ea8d60c15dfd596949 with result from task 778
parsl.dataflow.dflow DEBUG: Task 778 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 779 has memoization hash d9487385c26ba2f77922763e8492e5ab
parsl.dataflow.memoization INFO: Task 779 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 1033
parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f4e29f9e-43e2-4fe3-83d6-62dc3727a811_calibrate_946072_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f4e29f9e-43e2-4fe3-83d6-62dc3727a811_calibrate_946072_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/bfe2cf0d-0948-40a8-a6c2-0957ef50f3b8_characterizeImage_946072_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/bfe2cf0d-0948-40a8-a6c2-0957ef50f3b8_characterizeImage_946072_58.stderr
parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2124559503ccacd4ea79b25188bcd8 with result from task 554
parsl.dataflow.dflow DEBUG: Task 554 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4d9cb6ca-b335-4d37-b003-e5f22269b9b6_calibrate_946074_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4d9cb6ca-b335-4d37-b003-e5f22269b9b6_calibrate_946074_22.stderr
parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5e9564c21b43c6b90bbaa48cc52c749 with result from task 988
parsl.dataflow.dflow DEBUG: Task 988 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 8ac8da5edb286b7abc768b6757a43905
parsl.dataflow.memoization INFO: Task 989 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 1034
parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01440847-6532-4403-add2-85f29a1ea485_calibrate_946074_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01440847-6532-4403-add2-85f29a1ea485_calibrate_946074_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/545e7068-16ba-400c-a9e7-0445cc879021_characterizeImage_946074_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/545e7068-16ba-400c-a9e7-0445cc879021_characterizeImage_946074_23.stderr
parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d57dac4ff126f564cba80846409b99e with result from task 550
parsl.dataflow.dflow DEBUG: Task 550 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 551 has memoization hash c52035086ecd4b512626a84bb8065f32
parsl.dataflow.memoization INFO: Task 551 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 1035
parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/52813176-c6d2-4ff2-9a54-be23d48dfd53_calibrate_946181_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/52813176-c6d2-4ff2-9a54-be23d48dfd53_calibrate_946181_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d93100ac-38b9-492b-8b50-40f0f35fdb08_characterizeImage_946181_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d93100ac-38b9-492b-8b50-40f0f35fdb08_characterizeImage_946181_30.stderr
parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5263371d2e9e913e55d31bc596bc274 with result from task 769
parsl.dataflow.dflow DEBUG: Task 769 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 770 has memoization hash d8b7dc20309ed6a662d6c9a37e9ea719
parsl.dataflow.memoization INFO: Task 770 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 1036
parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cd03b765-a0e0-4aa4-9ce2-a722d1d82aa6_calibrate_946073_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cd03b765-a0e0-4aa4-9ce2-a722d1d82aa6_calibrate_946073_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2b86150-4836-431b-93b4-cd75a2b65cdd_characterizeImage_946073_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2b86150-4836-431b-93b4-cd75a2b65cdd_characterizeImage_946073_41.stderr
parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf63a538d96aa2bb64de4a9d63a7b0ce with result from task 781
parsl.dataflow.dflow DEBUG: Task 781 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 861d1addf89441d561b51dc9f4d9093a
parsl.dataflow.memoization INFO: Task 782 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 1037
parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/e22a380b-1892-4781-a4dd-f09054d1875e_calibrate_946073_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/e22a380b-1892-4781-a4dd-f09054d1875e_calibrate_946073_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c5b02300-6c56-45cf-9fe6-7b85222f29c3_characterizeImage_946073_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/c5b02300-6c56-45cf-9fe6-7b85222f29c3_characterizeImage_946073_46.stderr
parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec1a8f32ae4109eb6af8186e84ad169 with result from task 338
parsl.dataflow.dflow DEBUG: Task 338 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d20d7954-b831-46ed-a981-8eecee73c91e_calibrate_946181_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d20d7954-b831-46ed-a981-8eecee73c91e_calibrate_946181_13.stderr
parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5afdcdb0f9b95a01e895e3f49ccba598 with result from task 328
parsl.dataflow.dflow DEBUG: Task 328 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 7d3c899d19f28b22e64cff16c0ad8d56
parsl.dataflow.memoization INFO: Task 329 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 1038
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6e5c9724-8caf-4c19-8d13-c59a540ca79f_calibrate_946073_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6e5c9724-8caf-4c19-8d13-c59a540ca79f_calibrate_946073_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b3d3a648-d1f0-4fb4-8ab9-b9d66b3e9f1a_characterizeImage_946073_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b3d3a648-d1f0-4fb4-8ab9-b9d66b3e9f1a_characterizeImage_946073_15.stderr
parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9c0bd85e2fa044a525f196bf8bbfd19 with result from task 845
parsl.dataflow.dflow DEBUG: Task 845 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/19759d27-e92e-4a9f-a9c4-d935eb96c506_calibrate_946181_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/19759d27-e92e-4a9f-a9c4-d935eb96c506_calibrate_946181_27.stderr
parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 957a67948448dd2a2d9326ea15697c37 with result from task 836
parsl.dataflow.dflow DEBUG: Task 836 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d7890866-5e0e-47a8-9b5a-1c6698bf09c9_calibrate_946074_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d7890866-5e0e-47a8-9b5a-1c6698bf09c9_calibrate_946074_32.stderr
parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5d16afd46c672edad939b76efbfdc64 with result from task 763
parsl.dataflow.dflow DEBUG: Task 763 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 764 has memoization hash a81723cc0990eaba76ff2e1cc5b00798
parsl.dataflow.memoization INFO: Task 764 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 1039
parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/593be702-ff72-4014-8c52-e7a3199905ca_calibrate_946073_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/593be702-ff72-4014-8c52-e7a3199905ca_calibrate_946073_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1c316c6-307a-4100-b4bb-ba1de4a3c870_characterizeImage_946073_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1c316c6-307a-4100-b4bb-ba1de4a3c870_characterizeImage_946073_38.stderr
parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 328b83c9e04ef73a999ceca91dc3312a with result from task 898
parsl.dataflow.dflow DEBUG: Task 898 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 899 has memoization hash cc3e60583c7435ebffae3d5c4732cb62
parsl.dataflow.memoization INFO: Task 899 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 1040
parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d012d78f-4308-4e62-92af-07bcdaafc57e_calibrate_946074_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d012d78f-4308-4e62-92af-07bcdaafc57e_calibrate_946074_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ca58d1d-8ce7-4331-837b-2356ae6b53f4_characterizeImage_946074_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/8ca58d1d-8ce7-4331-837b-2356ae6b53f4_characterizeImage_946074_58.stderr
parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b074ebd771c10f32f54835f9fc7115b with result from task 787
parsl.dataflow.dflow DEBUG: Task 787 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 81a0577e86ba246942e8c31c55fd9707
parsl.dataflow.memoization INFO: Task 788 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 1041
parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a4b0d233-2d96-4fcd-bc1f-423ecdacbe8d_calibrate_946073_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a4b0d233-2d96-4fcd-bc1f-423ecdacbe8d_calibrate_946073_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/2c62f3d0-f4b7-4b00-ac9b-4fa39d84b178_characterizeImage_946073_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/2c62f3d0-f4b7-4b00-ac9b-4fa39d84b178_characterizeImage_946073_33.stderr
parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af1c38953920e812f7221613e8385bc9 with result from task 772
parsl.dataflow.dflow DEBUG: Task 772 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 302be1554056d7be6dc0d06f504cdabe
parsl.dataflow.memoization INFO: Task 773 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 1042
parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93d2e842-adda-41e2-853c-6f241cb8efa6_calibrate_946073_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93d2e842-adda-41e2-853c-6f241cb8efa6_calibrate_946073_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d4b80e25-25a5-4296-8316-ec79837b6643_characterizeImage_946073_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d4b80e25-25a5-4296-8316-ec79837b6643_characterizeImage_946073_58.stderr
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4418640ac4b0e79b2353f9d1c9f5977 with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4f1c90c5-1609-4753-ac29-4cfc77286901_calibrate_946181_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4f1c90c5-1609-4753-ac29-4cfc77286901_calibrate_946181_58.stderr
parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe3decec0c9649cabe2a4c74f55e9d86 with result from task 460
parsl.dataflow.dflow DEBUG: Task 460 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 461 has memoization hash f5a3557bc6cb9e95a717d21fb57684c5
parsl.dataflow.memoization INFO: Task 461 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a43380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 1043
parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8ac11115-0041-4c46-9ecd-2f6361230c96_calibrate_946182_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8ac11115-0041-4c46-9ecd-2f6361230c96_calibrate_946182_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5f5f8fbd-95e4-41fe-896f-7f9a54da4fc9_characterizeImage_946182_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/5f5f8fbd-95e4-41fe-896f-7f9a54da4fc9_characterizeImage_946182_9.stderr
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04ae2711da8f7ae6f608a27c7684dad1 with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4586958a-1864-4416-8bb0-25d79f7da970_calibrate_946073_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4586958a-1864-4416-8bb0-25d79f7da970_calibrate_946073_23.stderr
parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ec93e9ada2630e81e4937590b0f95f8 with result from task 563
parsl.dataflow.dflow DEBUG: Task 563 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/588e707c-3d87-46a1-9b9e-a0d76ce608d2_calibrate_946074_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/588e707c-3d87-46a1-9b9e-a0d76ce608d2_calibrate_946074_30.stderr
parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 978af61145fe565139a3e47dcf758c9f with result from task 961
parsl.dataflow.dflow DEBUG: Task 961 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 962 has memoization hash e2ce85b72f01f2b55f23b53077b9d4d6
parsl.dataflow.memoization INFO: Task 962 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 1044
parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e4d30549-bdbd-49a2-ae9f-9ff79dc49564_calibrate_946074_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e4d30549-bdbd-49a2-ae9f-9ff79dc49564_calibrate_946074_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7f2b2b44-7d9e-4a51-86e8-dd0225cd5f5e_characterizeImage_946074_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7f2b2b44-7d9e-4a51-86e8-dd0225cd5f5e_characterizeImage_946074_29.stderr
parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5f8c2c2a141de4ff9e9907725a56dfc with result from task 392
parsl.dataflow.dflow DEBUG: Task 392 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/5f032f64-4247-472f-819b-bb41b788fc40_calibrate_946072_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/5f032f64-4247-472f-819b-bb41b788fc40_calibrate_946072_21.stderr
parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca4bab1d8dd76b3b599c709cde8dde0f with result from task 664
parsl.dataflow.dflow DEBUG: Task 664 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 4cec15b912e634f207cecaae80ee5d82
parsl.dataflow.memoization INFO: Task 665 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308934900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 1045
parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18ec8a66-73f0-4d90-919d-eda29dccf0ae_calibrate_946180_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18ec8a66-73f0-4d90-919d-eda29dccf0ae_calibrate_946180_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/68b27ae8-b2ee-483a-a27a-7f8fd208a0bf_characterizeImage_946180_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/68b27ae8-b2ee-483a-a27a-7f8fd208a0bf_characterizeImage_946180_11.stderr
parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 299cbc1d254aa5db843646f24d4c3888 with result from task 944
parsl.dataflow.dflow DEBUG: Task 944 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/752361c9-eeab-44b8-b703-3ae3e964a122_calibrate_946074_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/752361c9-eeab-44b8-b703-3ae3e964a122_calibrate_946074_42.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 973cd8098a52fc6292109bafacb96d9e with result from task 470
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 205
parsl.dataflow.strategy DEBUG: Executor multi has 205 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow DEBUG: Task 470 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b85c0636-df72-4806-a8ad-d6918670f3b4_calibrate_946182_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b85c0636-df72-4806-a8ad-d6918670f3b4_calibrate_946182_35.stderr
parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0d001dc339f66360b97bdd9df1bd0ea with result from task 1006
parsl.dataflow.dflow DEBUG: Task 1006 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash e7502343cc13949c5df26e7f70e6dad2
parsl.dataflow.memoization INFO: Task 1007 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1046
parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/060134e8-b9d8-4db6-adce-ab37b64e80a1_calibrate_946074_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/060134e8-b9d8-4db6-adce-ab37b64e80a1_calibrate_946074_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7efea9d2-c122-454d-8426-ad270c23ac9b_characterizeImage_946074_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/7efea9d2-c122-454d-8426-ad270c23ac9b_characterizeImage_946074_25.stderr
parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4af90318d69a9a576b0a4478bc8b4d95 with result from task 925
parsl.dataflow.dflow DEBUG: Task 925 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 2f033f503111120f8d800dad568f5941
parsl.dataflow.memoization INFO: Task 926 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 1047
parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4e43d4ae-38d5-4413-b364-2db2cf476b1e_calibrate_946073_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4e43d4ae-38d5-4413-b364-2db2cf476b1e_calibrate_946073_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/94223191-96fd-4ab5-92c7-853383b7070c_characterizeImage_946073_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/94223191-96fd-4ab5-92c7-853383b7070c_characterizeImage_946073_43.stderr
parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2cf57bdd73d35dcaa06749adcdd3c6de with result from task 472
parsl.dataflow.dflow DEBUG: Task 472 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 1d4b35d705db499e36af97ada0c1ac89
parsl.dataflow.memoization INFO: Task 473 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a74040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 1048
parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/035b0805-aac8-4b83-99d1-5c2784260fff_calibrate_946073_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/035b0805-aac8-4b83-99d1-5c2784260fff_calibrate_946073_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1725caa-c8bc-447f-ac63-cc399b0d459c_characterizeImage_946073_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1725caa-c8bc-447f-ac63-cc399b0d459c_characterizeImage_946073_62.stderr
parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d824520f6218fef84a15ba1ad949e00 with result from task 920
parsl.dataflow.dflow DEBUG: Task 920 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e86340f8-c2eb-46fd-bd7f-284fefca75f8_calibrate_946074_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e86340f8-c2eb-46fd-bd7f-284fefca75f8_calibrate_946074_19.stderr
parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 846e1941fc71b274da7ecc949b5c262a with result from task 886
parsl.dataflow.dflow DEBUG: Task 886 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 887 has memoization hash bcadf0bd0b9d25e49d78f9a0c1412f00
parsl.dataflow.memoization INFO: Task 887 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 1049
parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0da64d9a-8739-406b-b17c-fab7a4803f90_calibrate_946181_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0da64d9a-8739-406b-b17c-fab7a4803f90_calibrate_946181_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6dd8b374-037c-47b7-ae85-1c91a7974fb5_characterizeImage_946181_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6dd8b374-037c-47b7-ae85-1c91a7974fb5_characterizeImage_946181_48.stderr
parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d6045993b2b2c6be27215101b9d821f with result from task 833
parsl.dataflow.dflow DEBUG: Task 833 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cb66dfb-acdb-4f70-b1e7-74d0596c9afe_calibrate_946182_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cb66dfb-acdb-4f70-b1e7-74d0596c9afe_calibrate_946182_1.stderr
parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 523b59f4d025193dc802b85c7cf83668 with result from task 518
parsl.dataflow.dflow DEBUG: Task 518 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b30ffcaa-230e-4390-bae0-571ffaec8ff5_calibrate_946074_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b30ffcaa-230e-4390-bae0-571ffaec8ff5_calibrate_946074_3.stderr
parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 015a6b8584e256ff329ae274a3660d01 with result from task 917
parsl.dataflow.dflow DEBUG: Task 917 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1ca10c20-af16-4292-ac38-2b06ed3ee94e_calibrate_946181_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1ca10c20-af16-4292-ac38-2b06ed3ee94e_calibrate_946181_9.stderr
parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 674b4e469285717323bcf8585980ce3c with result from task 932
parsl.dataflow.dflow DEBUG: Task 932 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/471e3a08-9547-47d9-ab73-caf47c73ab1c_calibrate_946073_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/471e3a08-9547-47d9-ab73-caf47c73ab1c_calibrate_946073_53.stderr
parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a28fdf103b1c2c4dac1b924550b9c582 with result from task 707
parsl.dataflow.dflow DEBUG: Task 707 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/f9f90c98-f857-4d02-99a2-8cba5ea3eb31_calibrate_946074_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/f9f90c98-f857-4d02-99a2-8cba5ea3eb31_calibrate_946074_34.stderr
parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d4b7c05233b7692d820c889372b7da7 with result from task 1114
parsl.dataflow.dflow DEBUG: Task 1114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 7a79af68541756d188c0c758b198d9cb
parsl.dataflow.memoization INFO: Task 1115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308735f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1050
parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/578584c8-694a-4b9f-a8b7-9fc8581bb936_calibrate_946181_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/578584c8-694a-4b9f-a8b7-9fc8581bb936_calibrate_946181_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/357abdff-b8be-4bfa-a2be-dca6fd5e322e_characterizeImage_946181_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/357abdff-b8be-4bfa-a2be-dca6fd5e322e_characterizeImage_946181_6.stderr
parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 175b47381e49182cb85e820d943e3138 with result from task 1031
parsl.dataflow.dflow DEBUG: Task 1031 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2fc0b3fe-0323-490f-8168-78cf2e55a744_calibrate_946182_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2fc0b3fe-0323-490f-8168-78cf2e55a744_calibrate_946182_55.stderr
parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee4cd158c1253740da39e5dc2b4162e6 with result from task 796
parsl.dataflow.dflow DEBUG: Task 796 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 150ccbdb6f51a34f42e967e221503981
parsl.dataflow.memoization INFO: Task 797 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 1051
parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f9ae6010-d9c6-4855-90fa-dd2fc368ba5a_calibrate_946073_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f9ae6010-d9c6-4855-90fa-dd2fc368ba5a_calibrate_946073_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e6751cb4-f578-485f-8971-804449eace20_characterizeImage_946073_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e6751cb4-f578-485f-8971-804449eace20_characterizeImage_946073_55.stderr
parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32bdb0a111a577e3ae3140d6c1ac8afb with result from task 532
parsl.dataflow.dflow DEBUG: Task 532 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 4db2ac0953eb5dbafdf8330650d56287
parsl.dataflow.memoization INFO: Task 533 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 1052
parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b6919ec8-5d28-47df-9d01-34780ae40729_calibrate_946182_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b6919ec8-5d28-47df-9d01-34780ae40729_calibrate_946182_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2b7cc1f2-c802-4055-ab68-8cc8480fa471_characterizeImage_946182_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/2b7cc1f2-c802-4055-ab68-8cc8480fa471_characterizeImage_946182_54.stderr
parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6309d088cc2c0b18d67daf57b0535738 with result from task 982
parsl.dataflow.dflow DEBUG: Task 982 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 064664868ad9bb641419899992849501
parsl.dataflow.memoization INFO: Task 983 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 1053
parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d3835bac-ac8c-4f49-bd6a-01eb7a1192ea_calibrate_946073_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d3835bac-ac8c-4f49-bd6a-01eb7a1192ea_calibrate_946073_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e1b08152-e56b-4ed7-913e-164bb11a71ec_characterizeImage_946073_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e1b08152-e56b-4ed7-913e-164bb11a71ec_characterizeImage_946073_57.stderr
parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 858c9ab7638285dec58910a7f6f9e194 with result from task 953
parsl.dataflow.dflow DEBUG: Task 953 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d561014a-854a-43e1-ae5f-6e061f1375c4_calibrate_946074_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d561014a-854a-43e1-ae5f-6e061f1375c4_calibrate_946074_57.stderr
parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5a692480244f35eade41d5b3083aacb1 with result from task 344
parsl.dataflow.dflow DEBUG: Task 344 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d886fde2-5cda-45aa-84e3-0b3c3197e6fa_calibrate_946181_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d886fde2-5cda-45aa-84e3-0b3c3197e6fa_calibrate_946181_21.stderr
parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22bbce3ba8df38b408c4827646bb3b79 with result from task 1109
parsl.dataflow.dflow DEBUG: Task 1109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4560f55a-3ee9-4210-8e4f-55c057f060ac_calibrate_946074_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4560f55a-3ee9-4210-8e4f-55c057f060ac_calibrate_946074_55.stderr
parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4e9561e8fb858dde147c4b1651ece6d with result from task 419
parsl.dataflow.dflow DEBUG: Task 419 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/361e4c74-c378-4aa7-98ff-521d1bd64e87_calibrate_946072_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/361e4c74-c378-4aa7-98ff-521d1bd64e87_calibrate_946072_54.stderr
parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16354dec216b4cce6a3c7febf51adb79 with result from task 340
parsl.dataflow.dflow DEBUG: Task 340 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 8616fc9ff5079f6eea12a5323e6bd762
parsl.dataflow.memoization INFO: Task 341 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089eb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 1054
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f288bedf-3091-4b46-ba03-cadc3fb3efc9_calibrate_946180_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f288bedf-3091-4b46-ba03-cadc3fb3efc9_calibrate_946180_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fcb7cd7c-ee47-4860-9f30-082eb7bb18be_characterizeImage_946180_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/fcb7cd7c-ee47-4860-9f30-082eb7bb18be_characterizeImage_946180_55.stderr
parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f0cb5f3e306b7c373c214a6fe6c5274 with result from task 605
parsl.dataflow.dflow DEBUG: Task 605 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/08b593b2-b065-4eea-b38f-9b501d971cd5_calibrate_946072_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/08b593b2-b065-4eea-b38f-9b501d971cd5_calibrate_946072_23.stderr
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 69e3a83c8a9abde7327bb8c274490472 with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/74b8bf98-b91c-4261-8dc5-1dfa59e53813_calibrate_946072_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/74b8bf98-b91c-4261-8dc5-1dfa59e53813_calibrate_946072_44.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 192
parsl.dataflow.strategy DEBUG: Executor multi has 192 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9f39686d83645b226b33ecf7ffe6eac with result from task 320
parsl.dataflow.dflow DEBUG: Task 320 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/878e6bfb-62ca-4352-bcd9-4e19f3e3255a_calibrate_946182_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/878e6bfb-62ca-4352-bcd9-4e19f3e3255a_calibrate_946182_62.stderr
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 61ce131e92ea8e8bde43c1264bd56d7e with result from task 49
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 8c8e3ea410a6ed8ba73da16fe6294536
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f885e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 1055
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a562cde2-de88-4f4a-9fca-c56ecf7a2707_calibrate_946180_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a562cde2-de88-4f4a-9fca-c56ecf7a2707_calibrate_946180_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/908dcb9e-d306-402b-a708-6f4f8e1715e0_characterizeImage_946180_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/908dcb9e-d306-402b-a708-6f4f8e1715e0_characterizeImage_946180_30.stderr
parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac71e0027ea94e196d4a24be4a476a8 with result from task 680
parsl.dataflow.dflow DEBUG: Task 680 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/810c42b3-eed8-485e-bd47-d2ee86ef3365_calibrate_946072_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/810c42b3-eed8-485e-bd47-d2ee86ef3365_calibrate_946072_4.stderr
parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8425528aed7a7b098bfb96e7daca2b57 with result from task 710
parsl.dataflow.dflow DEBUG: Task 710 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fa51ac9a-b855-40d2-b426-bd0c9f76e7d2_calibrate_946180_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/fa51ac9a-b855-40d2-b426-bd0c9f76e7d2_calibrate_946180_4.stderr
parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2b1c1e1304570de9b6a0d3c9f1f45ad with result from task 985
parsl.dataflow.dflow DEBUG: Task 985 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 84e69c12528acbd8693a2f901d7558d5
parsl.dataflow.memoization INFO: Task 986 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 1056
parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7bb221ca-43d5-4b4b-a03b-2d88321c9297_calibrate_946073_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7bb221ca-43d5-4b4b-a03b-2d88321c9297_calibrate_946073_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7e51b755-51dc-45ab-a83f-92a56ad4743f_characterizeImage_946073_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/7e51b755-51dc-45ab-a83f-92a56ad4743f_characterizeImage_946073_25.stderr
parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f46f2cdb354d37b2ed5ea03d40c7cc1 with result from task 745
parsl.dataflow.dflow DEBUG: Task 745 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 746 has memoization hash f2e0e2ac293a95e3cc2a09debdbec9b4
parsl.dataflow.memoization INFO: Task 746 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 1057
parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/864dfb3e-c4c0-4ecc-80af-540c33b18869_calibrate_946180_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/864dfb3e-c4c0-4ecc-80af-540c33b18869_calibrate_946180_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d121fbfe-8999-4ed7-a292-24fee80d56b6_characterizeImage_946180_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/d121fbfe-8999-4ed7-a292-24fee80d56b6_characterizeImage_946180_34.stderr
parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb43ef040e20bf9c872aa6c0a72654b with result from task 896
parsl.dataflow.dflow DEBUG: Task 896 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3909df4e-a3e3-401b-b36b-fa8e04b4d47a_calibrate_946072_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/3909df4e-a3e3-401b-b36b-fa8e04b4d47a_calibrate_946072_31.stderr
parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0685ff97ec65a70b0f9f2d7e75d0acad with result from task 911
parsl.dataflow.dflow DEBUG: Task 911 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5c73d9f1-44c4-41a2-b5b5-30704252ca96_calibrate_946182_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5c73d9f1-44c4-41a2-b5b5-30704252ca96_calibrate_946182_49.stderr
parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cfa2a39852aef68d1329e977192b463d with result from task 251
parsl.dataflow.dflow DEBUG: Task 251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a0ececf2-e375-4a52-b7ff-f80b15a4dca7_calibrate_946073_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a0ececf2-e375-4a52-b7ff-f80b15a4dca7_calibrate_946073_16.stderr
parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c24bf40c47b3c113f227c922d27aceca with result from task 536
parsl.dataflow.dflow DEBUG: Task 536 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b39beabf-b6c8-4cff-acd1-f39fbf7a0dee_calibrate_946074_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b39beabf-b6c8-4cff-acd1-f39fbf7a0dee_calibrate_946074_41.stderr
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed5999b4257041680125ec3148460545 with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5e66cea8-687f-4dcc-8ea7-ff7edde1178e_calibrate_946182_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5e66cea8-687f-4dcc-8ea7-ff7edde1178e_calibrate_946182_21.stderr
parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e8c49a9108729e2a22e28966b79980a with result from task 224
parsl.dataflow.dflow DEBUG: Task 224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ff6a4915-b856-498c-858c-13428c36890e_calibrate_946181_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/ff6a4915-b856-498c-858c-13428c36890e_calibrate_946181_28.stderr
parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea55bd1ec95dd2ee4c2724bbc4a41d1 with result from task 650
parsl.dataflow.dflow DEBUG: Task 650 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7cb7f79a-ffd9-4ad4-8b96-d33533761b7e_calibrate_946072_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7cb7f79a-ffd9-4ad4-8b96-d33533761b7e_calibrate_946072_1.stderr
parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd5a4c120c273c3729aef25dc763ee55 with result from task 890
parsl.dataflow.dflow DEBUG: Task 890 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/350f2cfb-45d6-41cc-b83e-845204144cfb_calibrate_946182_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/350f2cfb-45d6-41cc-b83e-845204144cfb_calibrate_946182_42.stderr
parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d2c658d0ff684bab63fe179b3dd6430 with result from task 968
parsl.dataflow.dflow DEBUG: Task 968 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3753f76-3175-478d-aa36-e72031992688_calibrate_946073_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/c3753f76-3175-478d-aa36-e72031992688_calibrate_946073_22.stderr
parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9ba0572c46a4fa1af12b5ce80ccaba0 with result from task 808
parsl.dataflow.dflow DEBUG: Task 808 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 809 has memoization hash dad18012f38838aba22f902398e1d4ce
parsl.dataflow.memoization INFO: Task 809 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 1058
parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/34b84db4-b1f2-4b06-aa03-4f7a61656a6c_calibrate_946073_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/34b84db4-b1f2-4b06-aa03-4f7a61656a6c_calibrate_946073_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d68bb6e6-3a8b-4d13-96a2-7b5e1b7bc5f7_characterizeImage_946073_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/d68bb6e6-3a8b-4d13-96a2-7b5e1b7bc5f7_characterizeImage_946073_42.stderr
parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e8aa67185fd4a1b8f6841a2c62c39e2 with result from task 980
parsl.dataflow.dflow DEBUG: Task 980 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/04629f76-7c03-4392-a99b-fd4abf2973ab_calibrate_946074_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/04629f76-7c03-4392-a99b-fd4abf2973ab_calibrate_946074_43.stderr
parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c14b01b44934a9347950f1224f184c89 with result from task 668
parsl.dataflow.dflow DEBUG: Task 668 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7831dcaa-4a9c-40f9-88c9-826a982707e4_calibrate_946072_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/7831dcaa-4a9c-40f9-88c9-826a982707e4_calibrate_946072_17.stderr
parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7418b30961e4ed6e1a15a265573eb166 with result from task 805
parsl.dataflow.dflow DEBUG: Task 805 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 5111de21fc9dd82fe88216d9a6ead7a4
parsl.dataflow.memoization INFO: Task 806 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 1059
parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d7bb436e-b352-49ce-aaf2-5b085c8826c7_calibrate_946073_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d7bb436e-b352-49ce-aaf2-5b085c8826c7_calibrate_946073_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f770fc60-6862-41eb-bb98-a057f9606f5b_characterizeImage_946073_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f770fc60-6862-41eb-bb98-a057f9606f5b_characterizeImage_946073_37.stderr
parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff963189d0c194a5126324be7f1c155e with result from task 904
parsl.dataflow.dflow DEBUG: Task 904 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 02e8650d2cf435f4dfb7bb2a88c50ffd
parsl.dataflow.memoization INFO: Task 905 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 1060
parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/7025dd23-17a5-479e-a3f7-0977fded0e0a_calibrate_946074_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/7025dd23-17a5-479e-a3f7-0977fded0e0a_calibrate_946074_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3fc808d7-3a6d-46df-8f21-415d25874bba_characterizeImage_946074_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/3fc808d7-3a6d-46df-8f21-415d25874bba_characterizeImage_946074_12.stderr
parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f564aa939c016f0008aed9090566796 with result from task 736
parsl.dataflow.dflow DEBUG: Task 736 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 98cfa79a311bf91faeb0a8ccfa7f217d
parsl.dataflow.memoization INFO: Task 737 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 1061
parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/159d6e4f-5ac7-4da8-9c8f-486a3598b2c7_calibrate_946072_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/159d6e4f-5ac7-4da8-9c8f-486a3598b2c7_calibrate_946072_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6d65eb9e-220e-4fc1-8a16-592deb32ed81_characterizeImage_946072_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/6d65eb9e-220e-4fc1-8a16-592deb32ed81_characterizeImage_946072_30.stderr
parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afd8ba9bb7f8571ca98842d1fc25fd88 with result from task 799
parsl.dataflow.dflow DEBUG: Task 799 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 800 has memoization hash d52af34353239c920bd3994c0c6097ff
parsl.dataflow.memoization INFO: Task 800 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 1062
parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1ffc1a0d-c27b-4754-9652-fda00346189e_calibrate_946073_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1ffc1a0d-c27b-4754-9652-fda00346189e_calibrate_946073_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1aa29ad-d88f-4ddd-9706-497b4ab824a7_characterizeImage_946073_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/f1aa29ad-d88f-4ddd-9706-497b4ab824a7_characterizeImage_946073_30.stderr
parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06d6d55feae053499aa3092608707f95 with result from task 1085
parsl.dataflow.dflow DEBUG: Task 1085 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/03700e61-15af-4d96-9f6d-0d70bb34a1d0_calibrate_946072_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/03700e61-15af-4d96-9f6d-0d70bb34a1d0_calibrate_946072_6.stderr
parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1b66c91f5c9eacc260d192518d7747 with result from task 620
parsl.dataflow.dflow DEBUG: Task 620 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/479f0b6d-99f7-40ec-9ad9-f9e09f472b76_calibrate_946182_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/479f0b6d-99f7-40ec-9ad9-f9e09f472b76_calibrate_946182_8.stderr
2024-03-13 21:39:13 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:39:13 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:13 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80fd570218dfc6e23035ba131faded40 with result from task 734
parsl.dataflow.dflow DEBUG: Task 734 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/95b6a829-51af-4941-b6ad-7be0a86d37a8_calibrate_946180_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/95b6a829-51af-4941-b6ad-7be0a86d37a8_calibrate_946180_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68d22eddcf43952df068c02117b1e079 with result from task 557
parsl.dataflow.dflow DEBUG: Task 557 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8b8b5232-6d9d-49ec-84de-4771588553f4_calibrate_946182_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8b8b5232-6d9d-49ec-84de-4771588553f4_calibrate_946182_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:14 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a054d163be69b250850a29c60c18872f with result from task 1088
parsl.dataflow.dflow DEBUG: Task 1088 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9cb301c2-49df-4558-9ecb-8cc98167d31b_calibrate_946073_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/9cb301c2-49df-4558-9ecb-8cc98167d31b_calibrate_946073_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e66cc382658768d6483bf4cf09119aa7 with result from task 1036
parsl.dataflow.dflow DEBUG: Task 1036 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 24c334f4bf74a4c52813825cba6759d6
parsl.dataflow.memoization INFO: Task 1037 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1063
parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7dc7939f-69c5-48c5-8779-e3852fe9aebc_calibrate_946182_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7dc7939f-69c5-48c5-8779-e3852fe9aebc_calibrate_946182_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/855ae4f9-aba5-454e-a8be-d74bfcd70c42_characterizeImage_946182_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/855ae4f9-aba5-454e-a8be-d74bfcd70c42_characterizeImage_946182_29.stderr
parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db8c5e475e54e9e701073bb8786a4ed6 with result from task 802
parsl.dataflow.dflow DEBUG: Task 802 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 803 has memoization hash b00d53468cbb8c3f2b3e866e5427435b
parsl.dataflow.memoization INFO: Task 803 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 1064
parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/67bcda7e-910f-422d-ba26-8fdf7995b87f_calibrate_946180_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/67bcda7e-910f-422d-ba26-8fdf7995b87f_calibrate_946180_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ed26306a-a54b-4d75-b86f-2fa254d4ac5e_characterizeImage_946180_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ed26306a-a54b-4d75-b86f-2fa254d4ac5e_characterizeImage_946180_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da6ac0edf8c814d9298374c572c52ea0 with result from task 830
parsl.dataflow.dflow DEBUG: Task 830 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d80ec8ce-f312-4ffc-b4f3-9bf246fa3f8c_calibrate_946180_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/d80ec8ce-f312-4ffc-b4f3-9bf246fa3f8c_calibrate_946180_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef420cdda25e96efd5c5459450e7a7d3 with result from task 527
parsl.dataflow.dflow DEBUG: Task 527 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4afd7cc1-2c4c-4c27-bcc4-4738f86e50d2_calibrate_946074_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4afd7cc1-2c4c-4c27-bcc4-4738f86e50d2_calibrate_946074_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:15 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 385716257c7bdbe8df0534e6439cb947 with result from task 512
parsl.dataflow.dflow DEBUG: Task 512 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7a954a6-bb06-4055-a7d9-4fe3d4a0b712_calibrate_946073_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7a954a6-bb06-4055-a7d9-4fe3d4a0b712_calibrate_946073_17.stderr
parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2036fd0e1a4b228e29f019a9d1b710fb with result from task 317
parsl.dataflow.dflow DEBUG: Task 317 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c7ec35ce-273c-4dc1-8dea-fe7f289a6abf_calibrate_946180_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c7ec35ce-273c-4dc1-8dea-fe7f289a6abf_calibrate_946180_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b798c8d07528299543fd6240baa8bc20 with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3adfdaa3-3aeb-496c-a1c6-40d44bba918c_calibrate_946073_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/3adfdaa3-3aeb-496c-a1c6-40d44bba918c_calibrate_946073_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0269037074bf2e7c702279a49412ccd with result from task 965
parsl.dataflow.dflow DEBUG: Task 965 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b15535bd-a842-487c-9be2-3320e7ea0bcb_calibrate_946074_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b15535bd-a842-487c-9be2-3320e7ea0bcb_calibrate_946074_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:16 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea5fb4a9b2ad647a362f8c4d977c6d5d with result from task 541
parsl.dataflow.dflow DEBUG: Task 541 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 542 has memoization hash c612b129797a5000597fcad4cb05937a
parsl.dataflow.memoization INFO: Task 542 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 1065
parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee539c2f-43d9-4e10-af4d-a44a141a035f_calibrate_946072_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee539c2f-43d9-4e10-af4d-a44a141a035f_calibrate_946072_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/500cd511-2935-4b25-a5c4-fd88cada8e6d_characterizeImage_946072_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/500cd511-2935-4b25-a5c4-fd88cada8e6d_characterizeImage_946072_25.stderr
parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 099f779491164e98481852bb75167152 with result from task 640
parsl.dataflow.dflow DEBUG: Task 640 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 641 has memoization hash ff7c1ea8572485f18876d924ba70e94a
parsl.dataflow.memoization INFO: Task 641 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088eefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 1066
parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2edead1-db90-49fc-98c0-f8e0eb834577_calibrate_946182_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2edead1-db90-49fc-98c0-f8e0eb834577_calibrate_946182_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/25f521cd-ea68-49ed-8d38-0baf9053bea9_characterizeImage_946182_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/25f521cd-ea68-49ed-8d38-0baf9053bea9_characterizeImage_946182_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f46c81471dbe25929ac65820836555dc with result from task 1058
parsl.dataflow.dflow DEBUG: Task 1058 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac3e6519-3ffb-4b5f-8fba-970b90f67828_calibrate_946182_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac3e6519-3ffb-4b5f-8fba-970b90f67828_calibrate_946182_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3c2ef5fb808af988933a5e95b85a06 with result from task 824
parsl.dataflow.dflow DEBUG: Task 824 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/135d6146-f86b-452b-a29f-eb73ee15e4a5_calibrate_946180_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/135d6146-f86b-452b-a29f-eb73ee15e4a5_calibrate_946180_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:17 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3c57415374aa6fad6bb5103197a72dd with result from task 941
parsl.dataflow.dflow DEBUG: Task 941 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ca6135f8-d725-452d-823d-054bf59804ee_calibrate_946180_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ca6135f8-d725-452d-823d-054bf59804ee_calibrate_946180_23.stderr
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba4e79c10dcc501c286f3e9adc455145 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b1a0fd09-f8f8-44ac-9aa4-b26afde6fa09_calibrate_946074_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/b1a0fd09-f8f8-44ac-9aa4-b26afde6fa09_calibrate_946074_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:18 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:19 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 908b134de311e57afc78a32f8e221144 with result from task 1069
parsl.dataflow.dflow DEBUG: Task 1069 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 8c99549897ad5f9a8f323e0ee20b9ed5
parsl.dataflow.memoization INFO: Task 1070 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1067
parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/887ee751-5044-4a80-9288-f0720a436359_calibrate_946074_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/887ee751-5044-4a80-9288-f0720a436359_calibrate_946074_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/78166bed-694c-4733-b7c1-19e01085530a_characterizeImage_946074_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/78166bed-694c-4733-b7c1-19e01085530a_characterizeImage_946074_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a650681b5eb40f552096f8134b69fe7c with result from task 302
parsl.dataflow.dflow DEBUG: Task 302 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a44829fc-dc99-4817-acf5-79f07f8c595d_calibrate_946182_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a44829fc-dc99-4817-acf5-79f07f8c595d_calibrate_946182_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:20 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23d8cf72ee3ec41be124cf4e0d49bdb2 with result from task 884
parsl.dataflow.dflow DEBUG: Task 884 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b1cc193b-7260-494a-a241-dae6971d26f5_calibrate_946180_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b1cc193b-7260-494a-a241-dae6971d26f5_calibrate_946180_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1ccb9750e1aeaff8842bb94ae47430 with result from task 415
parsl.dataflow.dflow DEBUG: Task 415 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 416 has memoization hash bfc19122f8f8897e916625b73ac1a6f8
parsl.dataflow.memoization INFO: Task 416 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 1068
parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee78d8ea-e3f6-45b8-a715-13f0148f0bcb_calibrate_946180_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee78d8ea-e3f6-45b8-a715-13f0148f0bcb_calibrate_946180_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/02211b09-f547-4bbc-afd1-bcafa89864c4_characterizeImage_946180_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/02211b09-f547-4bbc-afd1-bcafa89864c4_characterizeImage_946180_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 358b3bd51dd34f16be921b776560ebac with result from task 784
parsl.dataflow.dflow DEBUG: Task 784 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 785 has memoization hash fe504f5583d853b56aebbe45d44683c4
parsl.dataflow.memoization INFO: Task 785 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 1069
parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7dcdaa85-56bb-4233-a033-7360839b0876_calibrate_946181_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7dcdaa85-56bb-4233-a033-7360839b0876_calibrate_946181_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd5d9424-be8d-4d4d-8ac5-c3a75b32c9af_characterizeImage_946181_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/fd5d9424-be8d-4d4d-8ac5-c3a75b32c9af_characterizeImage_946181_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:21 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43a44a1f3f063ea365f9f43c9228f2e3 with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc06821a-ef51-4654-9f13-38876dd495ef_calibrate_946180_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc06821a-ef51-4654-9f13-38876dd495ef_calibrate_946180_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc6339189b5ff6a87ac01ce2f1719c50 with result from task 947
parsl.dataflow.dflow DEBUG: Task 947 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6a7daec-466f-44cb-983d-e08921cf337c_calibrate_946074_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e6a7daec-466f-44cb-983d-e08921cf337c_calibrate_946074_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 336b7fd06c9707cc8558fb87da98a7ba with result from task 775
parsl.dataflow.dflow DEBUG: Task 775 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 776 has memoization hash f26736d4e600f57e8538ee25e6feea1c
parsl.dataflow.memoization INFO: Task 776 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 1070
parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c154fe25-a75f-434d-a8f3-3d5d093c44fe_calibrate_946181_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c154fe25-a75f-434d-a8f3-3d5d093c44fe_calibrate_946181_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/230bc3d2-a3ce-4cb5-914d-f6a79347ac3a_characterizeImage_946181_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/230bc3d2-a3ce-4cb5-914d-f6a79347ac3a_characterizeImage_946181_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99a10b0c45251efbe2a1c6066290c2bb with result from task 1003
parsl.dataflow.dflow DEBUG: Task 1003 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 6990964713985c71d5e2cf667b08fb8f
parsl.dataflow.memoization INFO: Task 1004 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1071
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5d368f03-d48b-41c0-b0e5-609b58220e40_calibrate_946074_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5d368f03-d48b-41c0-b0e5-609b58220e40_calibrate_946074_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/55825a35-38c3-4d06-8c9e-bd7776f15092_characterizeImage_946074_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/55825a35-38c3-4d06-8c9e-bd7776f15092_characterizeImage_946074_52.stderr
parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18913a8977c48db7f44cb93b61dbd44b with result from task 539
parsl.dataflow.dflow DEBUG: Task 539 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cf26aaa2-80d2-454d-9f43-08c3c0550bfc_calibrate_946180_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cf26aaa2-80d2-454d-9f43-08c3c0550bfc_calibrate_946180_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bddbe7dac7309eaa7ed32a63144236e6 with result from task 893
parsl.dataflow.dflow DEBUG: Task 893 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7286120-fdbd-4a9d-8157-50602ba3e535_calibrate_946073_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f7286120-fdbd-4a9d-8157-50602ba3e535_calibrate_946073_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:22 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c067c1cb6ac1b8249459db9371b6d10 with result from task 790
parsl.dataflow.dflow DEBUG: Task 790 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 234455eae36f59871a2d0e0d49e04bba
parsl.dataflow.memoization INFO: Task 791 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 1072
parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/45c2ba11-8f52-4167-a5f4-4713c91e8c04_calibrate_946074_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/45c2ba11-8f52-4167-a5f4-4713c91e8c04_calibrate_946074_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/bf2da13f-fe1b-4b8e-ad6e-ea81ae374a5d_characterizeImage_946074_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/bf2da13f-fe1b-4b8e-ad6e-ea81ae374a5d_characterizeImage_946074_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa4160e84a8c5c8530e7a75bc02d8e17 with result from task 994
parsl.dataflow.dflow DEBUG: Task 994 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 1970c36f22baa9019034fbdf381fa98b
parsl.dataflow.memoization INFO: Task 995 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1073
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/20a5ccb4-c6d2-4095-9ac7-b158a75bf30e_calibrate_946073_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/20a5ccb4-c6d2-4095-9ac7-b158a75bf30e_calibrate_946073_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/14035c60-fa81-4cc9-85a0-673224eabe8f_characterizeImage_946073_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/14035c60-fa81-4cc9-85a0-673224eabe8f_characterizeImage_946073_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:23 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:24 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df44f6f5097e89cef5af941e5af41b93 with result from task 1106
parsl.dataflow.dflow DEBUG: Task 1106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2f8e859-92a8-4329-b80b-34312a2d1543_calibrate_946182_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2f8e859-92a8-4329-b80b-34312a2d1543_calibrate_946182_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:24 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:24 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:24 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:24 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:24 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 156
parsl.dataflow.strategy DEBUG: Executor multi has 156 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 156
parsl.dataflow.strategy DEBUG: Executor multi has 156 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 149 try 0 failed
parsl.dataflow.dflow INFO: Task 149 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 0a42a3a93f3d5e3c14f3a16b25b9129a
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 1 launched on executor multi with executor id 1074
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stderr
parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c351699d96b455f2ca717eb565dff491 with result from task 500
parsl.dataflow.dflow DEBUG: Task 500 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a0953b81-dc92-45cb-8ac1-dc8b9a55696b_calibrate_946180_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a0953b81-dc92-45cb-8ac1-dc8b9a55696b_calibrate_946180_22.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a71a8f1db67f8d54ceb3a0510f4fd533 with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4da8fa31-7368-45ef-b9f6-2016b1ed6e01_calibrate_946074_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/4da8fa31-7368-45ef-b9f6-2016b1ed6e01_calibrate_946074_10.stderr
parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95cfa2196c020e04d4365f1a71a60edc with result from task 1112
parsl.dataflow.dflow DEBUG: Task 1112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0b6da01b-3278-4326-aaf4-3bdcae208a1f_calibrate_946074_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/0b6da01b-3278-4326-aaf4-3bdcae208a1f_calibrate_946074_36.stderr
parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5dcb01ff083da018fc35c5d5d1a7b4ab with result from task 677
parsl.dataflow.dflow DEBUG: Task 677 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8d9c4102-cedb-4b56-9989-7a32e0022c44_calibrate_946074_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/8d9c4102-cedb-4b56-9989-7a32e0022c44_calibrate_946074_1.stderr
parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed7ed5a4dfc6bda75c39f5bc24e45579 with result from task 565
parsl.dataflow.dflow DEBUG: Task 565 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 0d441d412baeb912f39c8c43df3bc275
parsl.dataflow.memoization INFO: Task 566 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308aa2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 1075
parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/4fa7167e-d945-4176-848b-fd97f45ff38d_calibrate_946182_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/4fa7167e-d945-4176-848b-fd97f45ff38d_calibrate_946182_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6a6e1609-8dfc-443c-929c-7eb6b97883d9_characterizeImage_946182_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/6a6e1609-8dfc-443c-929c-7eb6b97883d9_characterizeImage_946182_15.stderr
parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78c78c2b475a2bac82eeb90515dd1063 with result from task 190
parsl.dataflow.dflow DEBUG: Task 190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 89328f12f5079f1230f7439194eefde4
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 1076
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/37301de0-5228-4184-ab80-9f54dc7c410a_characterizeImage_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/37301de0-5228-4184-ab80-9f54dc7c410a_characterizeImage_946181_31.stderr
parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44d251afd58e584f8b5e30677baf2048 with result from task 449
parsl.dataflow.dflow DEBUG: Task 449 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/845c96a6-90ca-464e-933f-530e405b82d1_calibrate_946072_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/845c96a6-90ca-464e-933f-530e405b82d1_calibrate_946072_49.stderr
parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9487385c26ba2f77922763e8492e5ab with result from task 779
parsl.dataflow.dflow DEBUG: Task 779 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f4e29f9e-43e2-4fe3-83d6-62dc3727a811_calibrate_946072_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/f4e29f9e-43e2-4fe3-83d6-62dc3727a811_calibrate_946072_58.stderr
parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f85e7e08d6dc75f47373d4a4f2788316 with result from task 914
parsl.dataflow.dflow DEBUG: Task 914 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7cf62449-4fc8-4468-b6d4-294beb041157_calibrate_946073_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7cf62449-4fc8-4468-b6d4-294beb041157_calibrate_946073_35.stderr
parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3dec3780dfd3112e45476978ce7ebcc3 with result from task 281
parsl.dataflow.dflow DEBUG: Task 281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b84ef8ad-a546-45b6-b3a9-ed36c6fe8ce0_calibrate_946072_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/b84ef8ad-a546-45b6-b3a9-ed36c6fe8ce0_calibrate_946072_16.stderr
parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e65a00e658ad64af95b9778c2265020 with result from task 761
parsl.dataflow.dflow DEBUG: Task 761 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/46bd657a-ae11-4d97-b4cc-664a38fdcd1e_calibrate_946182_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/46bd657a-ae11-4d97-b4cc-664a38fdcd1e_calibrate_946182_52.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 147
parsl.dataflow.strategy DEBUG: Executor multi has 147 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1a06fc9bb97957d387c48ad3db1ea88 with result from task 629
parsl.dataflow.dflow DEBUG: Task 629 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5df4e049-6ad1-4099-9591-48d24d569664_calibrate_946181_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5df4e049-6ad1-4099-9591-48d24d569664_calibrate_946181_39.stderr
parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88e90dde7bc3b62e9f33b25118a9afca with result from task 509
parsl.dataflow.dflow DEBUG: Task 509 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0ede9ad5-a2a4-41e9-90d1-705944fe99d8_calibrate_946182_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0ede9ad5-a2a4-41e9-90d1-705944fe99d8_calibrate_946182_10.stderr
parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44f681f9a7d6c767c5c6c5c84706010f with result from task 748
parsl.dataflow.dflow DEBUG: Task 748 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 749 has memoization hash a8127fb8269199f233600471245c2c41
parsl.dataflow.memoization INFO: Task 749 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 1077
parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e6bf85af-e6c4-4156-bacf-3de785a9fd98_calibrate_946181_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e6bf85af-e6c4-4156-bacf-3de785a9fd98_calibrate_946181_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1555dc34-c4c4-4d0e-8236-af86be5b4c82_characterizeImage_946181_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1555dc34-c4c4-4d0e-8236-af86be5b4c82_characterizeImage_946181_20.stderr
parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9829f5ade50bdbec319a577423f3b63a with result from task 881
parsl.dataflow.dflow DEBUG: Task 881 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ac95aa6f-19ad-4942-93bc-74404920d9db_calibrate_946180_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ac95aa6f-19ad-4942-93bc-74404920d9db_calibrate_946180_14.stderr
parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f72db76c274d2bfd963e215d7e82f47 with result from task 305
parsl.dataflow.dflow DEBUG: Task 305 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a872d42b-6e02-4baa-a215-94c9f9130163_calibrate_946182_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/a872d42b-6e02-4baa-a215-94c9f9130163_calibrate_946182_4.stderr
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a8c06f25ada358c800fe2b8d62ae474 with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9cd3467e-b10a-4df7-bd14-bc5048fdcdd0_calibrate_946181_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/9cd3467e-b10a-4df7-bd14-bc5048fdcdd0_calibrate_946181_41.stderr
parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 306e8c604ce78bd7dfbb2790e942d8ad with result from task 959
parsl.dataflow.dflow DEBUG: Task 959 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/1331b609-d511-4661-9d12-c3492c42babe_calibrate_946180_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/1331b609-d511-4661-9d12-c3492c42babe_calibrate_946180_45.stderr
parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e30ae19cbe5503d8468c5439a60f0553 with result from task 1091
parsl.dataflow.dflow DEBUG: Task 1091 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/657b5de9-a944-4366-b4e2-60b145f2cf9d_calibrate_946181_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/657b5de9-a944-4366-b4e2-60b145f2cf9d_calibrate_946181_8.stderr
parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8ca3ac7bbf00e9471ed90f988df1855 with result from task 1046
parsl.dataflow.dflow DEBUG: Task 1046 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/dda3d422-a709-4b1e-b0fb-52824a435cb3_calibrate_946181_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/dda3d422-a709-4b1e-b0fb-52824a435cb3_calibrate_946181_40.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 139
parsl.dataflow.strategy DEBUG: Executor multi has 139 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b95c61fd9f825b3cde9703ad84b13ab with result from task 560
parsl.dataflow.dflow DEBUG: Task 560 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/984607d3-1958-4812-931a-a6319165f53f_calibrate_946181_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/984607d3-1958-4812-931a-a6319165f53f_calibrate_946181_33.stderr
parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0228db358f915e65541e55b7ef246e52 with result from task 751
parsl.dataflow.dflow DEBUG: Task 751 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 752 has memoization hash f3220e8f4255a8df36d231b233919f56
parsl.dataflow.memoization INFO: Task 752 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 1078
parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d2d5dc52-faae-4f89-8f3a-d8aa076d1d23_calibrate_946074_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d2d5dc52-faae-4f89-8f3a-d8aa076d1d23_calibrate_946074_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/0c8c49fe-93e0-4d9e-bee3-7cee27d0eff3_characterizeImage_946074_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/0c8c49fe-93e0-4d9e-bee3-7cee27d0eff3_characterizeImage_946074_49.stderr
parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae1c1e6e28cb5457028a6a6aeb4470d with result from task 1001
parsl.dataflow.dflow DEBUG: Task 1001 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d618ff84-681e-4663-a641-de755752358c_calibrate_946073_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d618ff84-681e-4663-a641-de755752358c_calibrate_946073_26.stderr
parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2979fed58c7a3a558954e25c67fd5f with result from task 859
parsl.dataflow.dflow DEBUG: Task 859 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 860 has memoization hash ca9415e5d64792d3e8566542f8a9c972
parsl.dataflow.memoization INFO: Task 860 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088094e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 1079
parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fb98e0f9-6c14-49f2-8c41-df1a50277d77_calibrate_946074_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fb98e0f9-6c14-49f2-8c41-df1a50277d77_calibrate_946074_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9daa037b-f4c0-44cc-a778-c2132f792500_characterizeImage_946074_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/9daa037b-f4c0-44cc-a778-c2132f792500_characterizeImage_946074_24.stderr
parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c85991f667e92eff4e1cb30947d9bf7 with result from task 299
parsl.dataflow.dflow DEBUG: Task 299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c10b0c38-2d8d-4d6e-8759-2bc12d8e4c3e_calibrate_946180_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c10b0c38-2d8d-4d6e-8759-2bc12d8e4c3e_calibrate_946180_52.stderr
parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fe6f2b1af4e54cb60b327d2359a0dd8 with result from task 497
parsl.dataflow.dflow DEBUG: Task 497 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c4efff05-24d5-45fb-8d78-50ea0dddf600_calibrate_946180_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c4efff05-24d5-45fb-8d78-50ea0dddf600_calibrate_946180_18.stderr
parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 861d1addf89441d561b51dc9f4d9093a with result from task 782
parsl.dataflow.dflow DEBUG: Task 782 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/e22a380b-1892-4781-a4dd-f09054d1875e_calibrate_946073_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/e22a380b-1892-4781-a4dd-f09054d1875e_calibrate_946073_46.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c38f0ee1b8d0871e6a64fe06c84804cc with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f40be76b-a8ea-4dd1-8532-875952591beb_calibrate_946182_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f40be76b-a8ea-4dd1-8532-875952591beb_calibrate_946182_51.stderr
parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a81723cc0990eaba76ff2e1cc5b00798 with result from task 764
parsl.dataflow.dflow DEBUG: Task 764 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/593be702-ff72-4014-8c52-e7a3199905ca_calibrate_946073_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/593be702-ff72-4014-8c52-e7a3199905ca_calibrate_946073_38.stderr
parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 785db8a9186ac7d3c0fc5f1fc41535de with result from task 1033
parsl.dataflow.dflow DEBUG: Task 1033 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 54c8476111b8572d5b932651e5602e53
parsl.dataflow.memoization INFO: Task 1034 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1080
parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/77dd17bf-1237-4331-a825-19b38545dfa9_calibrate_946181_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/77dd17bf-1237-4331-a825-19b38545dfa9_calibrate_946181_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/02077692-e0ad-48e3-911a-a976e8e566aa_characterizeImage_946181_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/02077692-e0ad-48e3-911a-a976e8e566aa_characterizeImage_946181_15.stderr
parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c94095be96a68f47f77c48f0512d673 with result from task 643
parsl.dataflow.dflow DEBUG: Task 643 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 472624522d75345ad33701987c7e4daa
parsl.dataflow.memoization INFO: Task 644 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 1081
parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4f43bb63-9d80-4486-b665-79e6a0d32b26_calibrate_946073_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4f43bb63-9d80-4486-b665-79e6a0d32b26_calibrate_946073_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e806b8e4-bcac-4be2-b202-89d458c64d00_characterizeImage_946073_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/e806b8e4-bcac-4be2-b202-89d458c64d00_characterizeImage_946073_29.stderr
parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc3e60583c7435ebffae3d5c4732cb62 with result from task 899
parsl.dataflow.dflow DEBUG: Task 899 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d012d78f-4308-4e62-92af-07bcdaafc57e_calibrate_946074_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d012d78f-4308-4e62-92af-07bcdaafc57e_calibrate_946074_58.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ba4b58d8b4b72de0d93aa500bd133b4 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b1911219-6ba6-45a1-99a0-2c6b1948b68d_calibrate_946073_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/b1911219-6ba6-45a1-99a0-2c6b1948b68d_calibrate_946073_6.stderr
parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d9e59571859a279d25d07527816dee0 with result from task 766
parsl.dataflow.dflow DEBUG: Task 766 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 84744e47cb2e5a850b5a996a8e79de66
parsl.dataflow.memoization INFO: Task 767 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 1082
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fe96cda2-d114-4cfe-b4b9-6457de059198_characterizeImage_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/fe96cda2-d114-4cfe-b4b9-6457de059198_characterizeImage_946182_31.stderr
parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f033f503111120f8d800dad568f5941 with result from task 926
parsl.dataflow.dflow DEBUG: Task 926 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4e43d4ae-38d5-4413-b364-2db2cf476b1e_calibrate_946073_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4e43d4ae-38d5-4413-b364-2db2cf476b1e_calibrate_946073_43.stderr
parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7502343cc13949c5df26e7f70e6dad2 with result from task 1007
parsl.dataflow.dflow DEBUG: Task 1007 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/060134e8-b9d8-4db6-adce-ab37b64e80a1_calibrate_946074_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/060134e8-b9d8-4db6-adce-ab37b64e80a1_calibrate_946074_25.stderr
parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9372d35ac69794534f2dceb5b8afbdc with result from task 1061
parsl.dataflow.dflow DEBUG: Task 1061 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0379e2ae-a736-4a23-8e1b-fd501ae4eaad_calibrate_946181_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0379e2ae-a736-4a23-8e1b-fd501ae4eaad_calibrate_946181_17.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 127
parsl.dataflow.strategy DEBUG: Executor multi has 127 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d6d51e4c5a9d3878758776c0d808580 with result from task 1018
parsl.dataflow.dflow DEBUG: Task 1018 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 3d29724824998bc007b71430226d69ff
parsl.dataflow.memoization INFO: Task 1019 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1083
parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/fa8c2dd2-592a-4202-8e8b-e01b31fb73cf_calibrate_946073_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/fa8c2dd2-592a-4202-8e8b-e01b31fb73cf_calibrate_946073_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2365734-2076-4518-833a-45f97b8c3245_characterizeImage_946073_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/b2365734-2076-4518-833a-45f97b8c3245_characterizeImage_946073_44.stderr
parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b141df7609bac62d94df53be0c9c94d with result from task 935
parsl.dataflow.dflow DEBUG: Task 935 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9c2912c9-1556-4601-87f3-ca7f33d97c57_calibrate_946182_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9c2912c9-1556-4601-87f3-ca7f33d97c57_calibrate_946182_3.stderr
parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 302be1554056d7be6dc0d06f504cdabe with result from task 773
parsl.dataflow.dflow DEBUG: Task 773 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93d2e842-adda-41e2-853c-6f241cb8efa6_calibrate_946073_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/93d2e842-adda-41e2-853c-6f241cb8efa6_calibrate_946073_58.stderr
parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5408d7e16478444d56542dad4658117 with result from task 239
parsl.dataflow.dflow DEBUG: Task 239 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/09036a4b-a5f6-45e6-9e18-c3b7c73be56d_calibrate_946181_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/09036a4b-a5f6-45e6-9e18-c3b7c73be56d_calibrate_946181_25.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc9781abfa6a60a4f4b73b7c505bda60 with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 3284ee5fc71d9daa5b106c0f5697736b
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 1084
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/90294a33-5006-4795-acdf-a2548c72df76_calibrate_946180_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/90294a33-5006-4795-acdf-a2548c72df76_calibrate_946180_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a0bccbb6-5e16-476d-8649-b85ad57f9e9a_characterizeImage_946180_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/a0bccbb6-5e16-476d-8649-b85ad57f9e9a_characterizeImage_946180_16.stderr
parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5a0d05b3b410ff3f5b39f1ac7be357 with result from task 595
parsl.dataflow.dflow DEBUG: Task 595 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 596 has memoization hash cd0ec86ae6f6f730e417a90e131420e0
parsl.dataflow.memoization INFO: Task 596 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 1085
parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2204fe00-3931-4bf5-a12e-bc4ace635dd4_calibrate_946182_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2204fe00-3931-4bf5-a12e-bc4ace635dd4_calibrate_946182_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0da657d1-1cdb-49c7-a997-5cbe1548b194_characterizeImage_946182_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/0da657d1-1cdb-49c7-a997-5cbe1548b194_characterizeImage_946182_18.stderr
parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 633ab14526aa68db954331bc700a6523 with result from task 1067
parsl.dataflow.dflow DEBUG: Task 1067 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/2cfbe8b0-5360-470b-a3d8-6b907d6fb9f7_calibrate_946180_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/2cfbe8b0-5360-470b-a3d8-6b907d6fb9f7_calibrate_946180_57.stderr
parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56f93ed7b96ab459dfcd86270ca234a0 with result from task 503
parsl.dataflow.dflow DEBUG: Task 503 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3f73d4f-a89e-43ec-b7c3-079f4dcb71aa_calibrate_946181_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f3f73d4f-a89e-43ec-b7c3-079f4dcb71aa_calibrate_946181_56.stderr
parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2a41aa67e7ab3f08ac2f6b0d5356a1e with result from task 407
parsl.dataflow.dflow DEBUG: Task 407 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac6cccd0-56e2-4399-94d5-21c74775ff95_calibrate_946182_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ac6cccd0-56e2-4399-94d5-21c74775ff95_calibrate_946182_48.stderr
parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e147f025288badb0163fcf6e56ce1c7a with result from task 754
parsl.dataflow.dflow DEBUG: Task 754 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 6a7b1a3ea26e7a00ebf7e39a2a170357
parsl.dataflow.memoization INFO: Task 755 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 1086
parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/388e51cc-967f-492c-a69a-b80e7401bbaa_calibrate_946181_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/388e51cc-967f-492c-a69a-b80e7401bbaa_calibrate_946181_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/260be758-8f5d-4838-9695-5eb9498d6852_characterizeImage_946181_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/260be758-8f5d-4838-9695-5eb9498d6852_characterizeImage_946181_1.stderr
parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a186830eaf21d187232f27d72216766b with result from task 548
parsl.dataflow.dflow DEBUG: Task 548 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4de0082e-2162-4501-b2be-ed153b21f2c8_calibrate_946181_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4de0082e-2162-4501-b2be-ed153b21f2c8_calibrate_946181_38.stderr
parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2f4808eea9108188dbc77f37fa9a219 with result from task 868
parsl.dataflow.dflow DEBUG: Task 868 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 869 has memoization hash 63ad2eb4ba8509041d7f0a0ff7481274
parsl.dataflow.memoization INFO: Task 869 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 1087
parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7a9bb4f4-3782-44ab-9c32-12c33546ea9a_calibrate_946182_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7a9bb4f4-3782-44ab-9c32-12c33546ea9a_calibrate_946182_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1d77c8d2-1be8-4085-ace9-47dbdd6d8207_characterizeImage_946182_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/1d77c8d2-1be8-4085-ace9-47dbdd6d8207_characterizeImage_946182_53.stderr
parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d798efaee12c56e249f70d1ae397aa8b with result from task 1073
parsl.dataflow.dflow DEBUG: Task 1073 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8920a4d0-8a2c-4ab4-90aa-e61d21d8f1d9_calibrate_946180_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8920a4d0-8a2c-4ab4-90aa-e61d21d8f1d9_calibrate_946180_35.stderr
parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e51899611fb21fbdc53be3150795255 with result from task 701
parsl.dataflow.dflow DEBUG: Task 701 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2caae7ac-3ce8-4b7d-965b-0b4c3805d08c_calibrate_946182_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2caae7ac-3ce8-4b7d-965b-0b4c3805d08c_calibrate_946182_6.stderr
parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac12ee75e6bbe45a40e33a4da2bbd05 with result from task 521
parsl.dataflow.dflow DEBUG: Task 521 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a72a750d-bf8b-46c4-a00b-b79480023c1c_calibrate_946073_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a72a750d-bf8b-46c4-a00b-b79480023c1c_calibrate_946073_24.stderr
parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ac8da5edb286b7abc768b6757a43905 with result from task 989
parsl.dataflow.dflow DEBUG: Task 989 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01440847-6532-4403-add2-85f29a1ea485_calibrate_946074_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/01440847-6532-4403-add2-85f29a1ea485_calibrate_946074_23.stderr
parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32ec5e51cdb5e2f8615ae731357f3bad with result from task 608
parsl.dataflow.dflow DEBUG: Task 608 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3f5195f9-0153-4eba-bb39-660310b03ff1_calibrate_946182_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3f5195f9-0153-4eba-bb39-660310b03ff1_calibrate_946182_45.stderr
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:45 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1fde992dc2c59fbf2d48d5f78704307 with result from task 524
parsl.dataflow.dflow DEBUG: Task 524 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/41518cc6-5c6d-4b71-ad17-a2ee250df8a3_calibrate_946181_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/41518cc6-5c6d-4b71-ad17-a2ee250df8a3_calibrate_946181_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0573a2325676be80015ab8c9d25fcb0 with result from task 659
parsl.dataflow.dflow DEBUG: Task 659 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3957b707-722c-4a58-b0ae-1e29025b4731_calibrate_946180_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/3957b707-722c-4a58-b0ae-1e29025b4731_calibrate_946180_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e1773192a982e201f141b802d8b8c93 with result from task 856
parsl.dataflow.dflow DEBUG: Task 856 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 790dc8ee46b2d68a2aee0f4f0641c42f
parsl.dataflow.memoization INFO: Task 857 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088091c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 1088
parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/66a4e4a7-be69-45d4-aa86-b825ec2422fb_calibrate_946182_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/66a4e4a7-be69-45d4-aa86-b825ec2422fb_calibrate_946182_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/feb6e222-72d7-4abe-8c57-548ebc4facbf_characterizeImage_946182_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/feb6e222-72d7-4abe-8c57-548ebc4facbf_characterizeImage_946182_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:46 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 75320902fe218fcec539d567b2c152b4 with result from task 506
parsl.dataflow.dflow DEBUG: Task 506 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5d471be8-0189-4b12-a638-86f5b63b806a_calibrate_946181_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/5d471be8-0189-4b12-a638-86f5b63b806a_calibrate_946181_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac70a6e67873121245294679eff3611e with result from task 850
parsl.dataflow.dflow DEBUG: Task 850 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 94f54b437ef48449d911c58e907ad111
parsl.dataflow.memoization INFO: Task 851 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 1089
parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7c439016-9cac-42aa-847d-526ddecae0b3_calibrate_946181_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7c439016-9cac-42aa-847d-526ddecae0b3_calibrate_946181_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/8a5704d7-4c31-49a1-84d6-ea94e0c4f8ea_characterizeImage_946181_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/8a5704d7-4c31-49a1-84d6-ea94e0c4f8ea_characterizeImage_946181_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a719e0a3393c403753ab8d20e472d495 with result from task 491
parsl.dataflow.dflow DEBUG: Task 491 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0016d22e-b80b-40a0-a952-8049f2a2855a_calibrate_946181_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0016d22e-b80b-40a0-a952-8049f2a2855a_calibrate_946181_29.stderr
parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2eade6c938151f583a0214a50a9f2b0 with result from task 1055
parsl.dataflow.dflow DEBUG: Task 1055 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0b2d1b90-14fd-46ab-88b9-4d3b96ad6dec_calibrate_946182_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/0b2d1b90-14fd-46ab-88b9-4d3b96ad6dec_calibrate_946182_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:47 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5b92cc854e36a16cc4a63886f3440f7 with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/31dbcf4d-cb01-44c9-8a9f-b446637bac95_calibrate_946180_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/31dbcf4d-cb01-44c9-8a9f-b446637bac95_calibrate_946180_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa1bbee9da6b6f7a7b58e76a114859cc with result from task 311
parsl.dataflow.dflow DEBUG: Task 311 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/67711a9a-4742-49d8-be33-66afbd3c898f_calibrate_946181_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/67711a9a-4742-49d8-be33-66afbd3c898f_calibrate_946181_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:48 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b74d83bd22ac8de88ca14001526696e with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/62f22ce3-54d4-47be-acee-fbab1bbb8fad_calibrate_946182_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/62f22ce3-54d4-47be-acee-fbab1bbb8fad_calibrate_946182_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f359b2a59ab391425f62b243c98ecf78 with result from task 997
parsl.dataflow.dflow DEBUG: Task 997 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 998 has memoization hash f68d28a0e4ff52cd8b5efeefe8b4d7d4
parsl.dataflow.memoization INFO: Task 998 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1090
parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/cd4f43c0-1a36-45d1-b7f3-d03c41d00162_calibrate_946182_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/cd4f43c0-1a36-45d1-b7f3-d03c41d00162_calibrate_946182_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b81ba4f0-4db3-42d6-9cd3-77eb55e108ee_characterizeImage_946182_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/b81ba4f0-4db3-42d6-9cd3-77eb55e108ee_characterizeImage_946182_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:49 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccf961a191094fbf2b090b8797fbc63c with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b42e89b2-0fa4-46d2-a85f-4767ee26c796_calibrate_946180_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b42e89b2-0fa4-46d2-a85f-4767ee26c796_calibrate_946180_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 150ccbdb6f51a34f42e967e221503981 with result from task 797
parsl.dataflow.dflow DEBUG: Task 797 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f9ae6010-d9c6-4855-90fa-dd2fc368ba5a_calibrate_946073_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/f9ae6010-d9c6-4855-90fa-dd2fc368ba5a_calibrate_946073_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:50 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dad18012f38838aba22f902398e1d4ce with result from task 809
parsl.dataflow.dflow DEBUG: Task 809 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/34b84db4-b1f2-4b06-aa03-4f7a61656a6c_calibrate_946073_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/34b84db4-b1f2-4b06-aa03-4f7a61656a6c_calibrate_946073_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98cfa79a311bf91faeb0a8ccfa7f217d with result from task 737
parsl.dataflow.dflow DEBUG: Task 737 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/159d6e4f-5ac7-4da8-9c8f-486a3598b2c7_calibrate_946072_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/159d6e4f-5ac7-4da8-9c8f-486a3598b2c7_calibrate_946072_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:51 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28843729de0cd32431569ff2ce5cd63e with result from task 1009
parsl.dataflow.dflow DEBUG: Task 1009 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash a489b0eee3a2c9da81199edc61645617
parsl.dataflow.memoization INFO: Task 1010 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1091
parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/88919732-ab87-4d2b-a0b0-c5c7b1aae111_calibrate_946180_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/88919732-ab87-4d2b-a0b0-c5c7b1aae111_calibrate_946180_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0afc4433-d75d-43db-b08e-d974935d7158_characterizeImage_946180_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/0afc4433-d75d-43db-b08e-d974935d7158_characterizeImage_946180_43.stderr
parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5025bd1b8880820082d3b57d0b569dd8 with result from task 1075
parsl.dataflow.dflow DEBUG: Task 1075 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 32b029a09c2197c7fc9a47692d77c3fb
parsl.dataflow.memoization INFO: Task 1076 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1092
parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee60e4e2-8b46-4b97-a3ce-142161cd027b_calibrate_946180_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee60e4e2-8b46-4b97-a3ce-142161cd027b_calibrate_946180_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e680f82d-4f07-4c8d-aa88-e649c7b35dda_characterizeImage_946180_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e680f82d-4f07-4c8d-aa88-e649c7b35dda_characterizeImage_946180_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d253cf418af3cd7cbc3a4f53ee9538ab with result from task 1082
parsl.dataflow.dflow DEBUG: Task 1082 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e30248a5-135e-46fd-bf87-8153bc95a7b3_calibrate_946182_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e30248a5-135e-46fd-bf87-8153bc95a7b3_calibrate_946182_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cec15b912e634f207cecaae80ee5d82 with result from task 665
parsl.dataflow.dflow DEBUG: Task 665 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18ec8a66-73f0-4d90-919d-eda29dccf0ae_calibrate_946180_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/18ec8a66-73f0-4d90-919d-eda29dccf0ae_calibrate_946180_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:52 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e1aff4edc07307a2bd0747bf3285378 with result from task 1103
parsl.dataflow.dflow DEBUG: Task 1103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/be2dc455-f43b-4534-b8d3-a7c1b751e4b3_calibrate_946180_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/be2dc455-f43b-4534-b8d3-a7c1b751e4b3_calibrate_946180_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4282fe70ed1118f3688a45c16e406a9e with result from task 956
parsl.dataflow.dflow DEBUG: Task 956 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/b1626dd8-fa6f-42d8-b4a9-6b1023371793_calibrate_946181_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/b1626dd8-fa6f-42d8-b4a9-6b1023371793_calibrate_946181_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4b35d705db499e36af97ada0c1ac89 with result from task 473
parsl.dataflow.dflow DEBUG: Task 473 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/035b0805-aac8-4b83-99d1-5c2784260fff_calibrate_946073_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/035b0805-aac8-4b83-99d1-5c2784260fff_calibrate_946073_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:53 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9644ab5c2e0b1e3293ea5abf29abfa8 with result from task 871
parsl.dataflow.dflow DEBUG: Task 871 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 55352615daddd78196cec6840dc04500
parsl.dataflow.memoization INFO: Task 872 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230880a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 1093
parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/de1eb473-1648-45d4-8bc5-02f719322845_calibrate_946182_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/de1eb473-1648-45d4-8bc5-02f719322845_calibrate_946182_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c9ecbca7-9abb-44ff-b88a-f79f3584af08_characterizeImage_946182_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/c9ecbca7-9abb-44ff-b88a-f79f3584af08_characterizeImage_946182_56.stderr
parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2fbcdc366777ddcff62f8715d526f700 with result from task 674
parsl.dataflow.dflow DEBUG: Task 674 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8bebe307-0614-47d1-9cf7-5b7d0e06833d_calibrate_946072_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/8bebe307-0614-47d1-9cf7-5b7d0e06833d_calibrate_946072_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 191 try 0 failed
parsl.dataflow.dflow INFO: Task 191 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 89328f12f5079f1230f7439194eefde4
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fcd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 1 launched on executor multi with executor id 1094
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7ed911597abb6271f01a447a3b09a7 with result from task 325
parsl.dataflow.dflow DEBUG: Task 325 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 326 has memoization hash addf2ff53397b0cdfa80e9918cc9a4c5
parsl.dataflow.memoization INFO: Task 326 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089ea660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 1095
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b5892b91-2e68-447d-b600-8d474adfe5ef_calibrate_946180_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b5892b91-2e68-447d-b600-8d474adfe5ef_calibrate_946180_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/27e87bb0-c388-42ad-a680-ee1af40cecec_characterizeImage_946180_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/27e87bb0-c388-42ad-a680-ee1af40cecec_characterizeImage_946180_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:54 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a8a94d7ffa5f17c60e6c709c24e201b3 with result from task 742
parsl.dataflow.dflow DEBUG: Task 742 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 3c8ba6666aed2dc0a48b5ffbfa5db362
parsl.dataflow.memoization INFO: Task 743 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 1096
parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5201bf2b-91c5-4b7b-8574-3912a502b7e8_calibrate_946074_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5201bf2b-91c5-4b7b-8574-3912a502b7e8_calibrate_946074_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/435ca46f-0b19-4837-910a-a1c38e47dff1_characterizeImage_946074_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/435ca46f-0b19-4837-910a-a1c38e47dff1_characterizeImage_946074_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5111de21fc9dd82fe88216d9a6ead7a4 with result from task 806
parsl.dataflow.dflow DEBUG: Task 806 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d7bb436e-b352-49ce-aaf2-5b085c8826c7_calibrate_946073_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d7bb436e-b352-49ce-aaf2-5b085c8826c7_calibrate_946073_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e868da1746ca8db922b7ad5e97789c53 with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/65c52be7-5b59-4aa9-99df-a98839f1b94f_calibrate_946180_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/65c52be7-5b59-4aa9-99df-a98839f1b94f_calibrate_946180_7.stderr
parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 455f837289a8fb9f102d56178d913411 with result from task 614
parsl.dataflow.dflow DEBUG: Task 614 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c3c52260-7935-47d0-bb09-a7bc02ee4f78_calibrate_946182_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/c3c52260-7935-47d0-bb09-a7bc02ee4f78_calibrate_946182_34.stderr
parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 291f09a0604de839c5455dc73a5669e9 with result from task 656
parsl.dataflow.dflow DEBUG: Task 656 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8599c38c-3a5b-4924-85d8-09e5434104b4_calibrate_946180_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/8599c38c-3a5b-4924-85d8-09e5434104b4_calibrate_946180_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7a89008cfabdd44e9100f6a0973b0a with result from task 292
parsl.dataflow.dflow DEBUG: Task 292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash b61921df4ad2d966c34df23f3a1116a1
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 1097
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a4c94f7-3f67-425a-bf00-b72f0f06a20a_characterizeImage_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a4c94f7-3f67-425a-bf00-b72f0f06a20a_characterizeImage_946073_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:55 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:39:56 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2e0e2ac293a95e3cc2a09debdbec9b4 with result from task 746
parsl.dataflow.dflow DEBUG: Task 746 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/864dfb3e-c4c0-4ecc-80af-540c33b18869_calibrate_946180_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/864dfb3e-c4c0-4ecc-80af-540c33b18869_calibrate_946180_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 92
parsl.dataflow.strategy DEBUG: Executor multi has 92 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 92
parsl.dataflow.strategy DEBUG: Executor multi has 92 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0869b5e036eb4233ccffbff147286666 with result from task 922
parsl.dataflow.dflow DEBUG: Task 922 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 5254f92ddc4ac52a3d9ae76a01c66c62
parsl.dataflow.memoization INFO: Task 923 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230884d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 1098
parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4effc77a-8dd4-409f-bc98-6dc13c41532f_calibrate_946181_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4effc77a-8dd4-409f-bc98-6dc13c41532f_calibrate_946181_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e96b017d-ad8a-4d87-a8c5-d4b83492789a_characterizeImage_946181_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/e96b017d-ad8a-4d87-a8c5-d4b83492789a_characterizeImage_946181_62.stderr
parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 535be5ca0d45d6348d805c9a380f04f1 with result from task 1021
parsl.dataflow.dflow DEBUG: Task 1021 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 37ebe3a33ccc407b227f8631f425147f
parsl.dataflow.memoization INFO: Task 1022 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1099
parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee3d7c93-5492-4d1b-8862-17c2a0bc2180_calibrate_946072_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee3d7c93-5492-4d1b-8862-17c2a0bc2180_calibrate_946072_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/7c22df4c-01fd-4908-8f74-a256378b4c40_characterizeImage_946072_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/7c22df4c-01fd-4908-8f74-a256378b4c40_characterizeImage_946072_7.stderr
parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44996f025bfcd4db35bb38b54addfc18 with result from task 290
parsl.dataflow.dflow DEBUG: Task 290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ebb1f066-f472-4f07-8e70-ba0cfa3fda34_calibrate_946180_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ebb1f066-f472-4f07-8e70-ba0cfa3fda34_calibrate_946180_44.stderr
parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8baa323e81eb5c626120e45630507a with result from task 689
parsl.dataflow.dflow DEBUG: Task 689 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c0982490-7c40-494f-9fec-971e865efc2e_calibrate_946180_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/c0982490-7c40-494f-9fec-971e865efc2e_calibrate_946180_26.stderr
parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d84b5640ad9a4e759949d5dcbf279d2d with result from task 545
parsl.dataflow.dflow DEBUG: Task 545 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a163f383-a097-425e-ad60-32917cfdb990_calibrate_946180_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a163f383-a097-425e-ad60-32917cfdb990_calibrate_946180_1.stderr
parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7da2c593a791623fe97e20134ba3c1f with result from task 757
parsl.dataflow.dflow DEBUG: Task 757 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 52378138bf2f96fe87bd474aeba24c3d
parsl.dataflow.memoization INFO: Task 758 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308976a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 1100
parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24c3c65e-a9aa-49d3-84ce-9945167a49b8_calibrate_946181_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24c3c65e-a9aa-49d3-84ce-9945167a49b8_calibrate_946181_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1a867483-4012-4c58-b03f-ec849a399645_characterizeImage_946181_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/1a867483-4012-4c58-b03f-ec849a399645_characterizeImage_946181_47.stderr
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0647c9d90bfee012e4e6c922265d4aa1 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34dda894-cc51-4efa-ae11-8f6a884bd5fb_calibrate_946181_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/34dda894-cc51-4efa-ae11-8f6a884bd5fb_calibrate_946181_61.stderr
parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84e69c12528acbd8693a2f901d7558d5 with result from task 986
parsl.dataflow.dflow DEBUG: Task 986 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7bb221ca-43d5-4b4b-a03b-2d88321c9297_calibrate_946073_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/7bb221ca-43d5-4b4b-a03b-2d88321c9297_calibrate_946073_25.stderr
parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18fe291ea7a568b1717db10dd7e6274c with result from task 272
parsl.dataflow.dflow DEBUG: Task 272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/9c199466-bbdf-4db0-9286-25cc8c36c549_calibrate_946180_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/9c199466-bbdf-4db0-9286-25cc8c36c549_calibrate_946180_5.stderr
parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb3d364cfebb00a56557da3b9e699c88 with result from task 1027
parsl.dataflow.dflow DEBUG: Task 1027 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 18f3194746e6ba4cf298b2f091147d7b
parsl.dataflow.memoization INFO: Task 1028 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1101
parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/bb21a9e6-75db-412b-896c-919ee9f5506b_calibrate_946182_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/bb21a9e6-75db-412b-896c-919ee9f5506b_calibrate_946182_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfc016be-9300-467c-9433-06e46804795b_characterizeImage_946182_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/cfc016be-9300-467c-9433-06e46804795b_characterizeImage_946182_37.stderr
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba83095a96e53a77aae3d322c61bac43 with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9afa961a-aec1-4c67-a0fa-c45ee48f6c0e_calibrate_946182_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/9afa961a-aec1-4c67-a0fa-c45ee48f6c0e_calibrate_946182_50.stderr
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51f122101375fbf047f426a627a9e93a with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ea6708c5-088f-4021-aaa0-67e04c80d352_calibrate_946182_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/ea6708c5-088f-4021-aaa0-67e04c80d352_calibrate_946182_19.stderr
parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35fb18251901972b1a52689ac3b8adfe with result from task 839
parsl.dataflow.dflow DEBUG: Task 839 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/da01e288-092c-449d-a722-fe4381713a43_calibrate_946182_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/da01e288-092c-449d-a722-fe4381713a43_calibrate_946182_40.stderr
parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e2ce85b72f01f2b55f23b53077b9d4d6 with result from task 962
parsl.dataflow.dflow DEBUG: Task 962 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e4d30549-bdbd-49a2-ae9f-9ff79dc49564_calibrate_946074_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/e4d30549-bdbd-49a2-ae9f-9ff79dc49564_calibrate_946074_29.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 82
parsl.dataflow.strategy DEBUG: Executor multi has 82 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8e3ea410a6ed8ba73da16fe6294536 with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a562cde2-de88-4f4a-9fca-c56ecf7a2707_calibrate_946180_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/a562cde2-de88-4f4a-9fca-c56ecf7a2707_calibrate_946180_30.stderr
parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4db2ac0953eb5dbafdf8330650d56287 with result from task 533
parsl.dataflow.dflow DEBUG: Task 533 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b6919ec8-5d28-47df-9d01-34780ae40729_calibrate_946182_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b6919ec8-5d28-47df-9d01-34780ae40729_calibrate_946182_54.stderr
parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f596b4f059e9c66aed10b695c05fdbed with result from task 865
parsl.dataflow.dflow DEBUG: Task 865 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 866 has memoization hash afb581f232491dd73aa68cbfc3d34c8d
parsl.dataflow.memoization INFO: Task 866 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308809b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 1102
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1eaf3599-d8e5-4445-a8ce-a3b9a754f620_calibrate_946181_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1eaf3599-d8e5-4445-a8ce-a3b9a754f620_calibrate_946181_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7e4b4116-f597-47b9-b881-cb6b5e7b5a2d_characterizeImage_946181_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/7e4b4116-f597-47b9-b881-cb6b5e7b5a2d_characterizeImage_946181_12.stderr
parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8b7dc20309ed6a662d6c9a37e9ea719 with result from task 770
parsl.dataflow.dflow DEBUG: Task 770 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cd03b765-a0e0-4aa4-9ce2-a722d1d82aa6_calibrate_946073_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/cd03b765-a0e0-4aa4-9ce2-a722d1d82aa6_calibrate_946073_41.stderr
parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5f40093c793f2a9f711795465de1211 with result from task 260
parsl.dataflow.dflow DEBUG: Task 260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b5e72b97-8a2e-4a66-8523-c510dae0638f_calibrate_946182_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/b5e72b97-8a2e-4a66-8523-c510dae0638f_calibrate_946182_30.stderr
parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b0a726c94bdc48c993bff4ec59f021e with result from task 853
parsl.dataflow.dflow DEBUG: Task 853 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 854 has memoization hash f74687cb0075684d187a4c29faf16502
parsl.dataflow.memoization INFO: Task 854 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308808ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 1103
parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c8e2125f-fa49-4b32-9596-0908f761edf4_calibrate_946181_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c8e2125f-fa49-4b32-9596-0908f761edf4_calibrate_946181_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d4533c05-6ac2-44d8-815c-acc89a6f318d_characterizeImage_946181_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/d4533c05-6ac2-44d8-815c-acc89a6f318d_characterizeImage_946181_42.stderr
parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c99549897ad5f9a8f323e0ee20b9ed5 with result from task 1070
parsl.dataflow.dflow DEBUG: Task 1070 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/887ee751-5044-4a80-9288-f0720a436359_calibrate_946074_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/887ee751-5044-4a80-9288-f0720a436359_calibrate_946074_62.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 77
parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b00d53468cbb8c3f2b3e866e5427435b with result from task 803
parsl.dataflow.dflow DEBUG: Task 803 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/67bcda7e-910f-422d-ba26-8fdf7995b87f_calibrate_946180_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/67bcda7e-910f-422d-ba26-8fdf7995b87f_calibrate_946180_13.stderr
parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f148c1a7ef17377522a55f7e86a76629 with result from task 287
parsl.dataflow.dflow DEBUG: Task 287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/503cac68-f898-409d-b803-48b9ea3fc2f9_calibrate_946072_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/503cac68-f898-409d-b803-48b9ea3fc2f9_calibrate_946072_29.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 767 try 0 failed
parsl.dataflow.dflow INFO: Task 767 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 84744e47cb2e5a850b5a996a8e79de66
parsl.dataflow.memoization INFO: Task 767 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308977380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 767 try 1 launched on executor multi with executor id 1104
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stderr
parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dfbaecd2b4beff681f15c8089bbf570d with result from task 425
parsl.dataflow.dflow DEBUG: Task 425 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7bf77f3d-4ac3-4109-80a8-c6df331d522a_calibrate_946182_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7bf77f3d-4ac3-4109-80a8-c6df331d522a_calibrate_946182_13.stderr
parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e98a59f28377a4282dc4649d87f7339a with result from task 739
parsl.dataflow.dflow DEBUG: Task 739 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 3a018d0b95140947f7fe6fb7af0769aa
parsl.dataflow.memoization INFO: Task 740 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308975760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 1105
parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64e0392e-5cb3-486e-b319-175e2c9f4da3_calibrate_946073_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64e0392e-5cb3-486e-b319-175e2c9f4da3_calibrate_946073_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/28f5fb91-bfa1-4bfd-843b-e56727af878f_characterizeImage_946073_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/28f5fb91-bfa1-4bfd-843b-e56727af878f_characterizeImage_946073_7.stderr
parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81a0577e86ba246942e8c31c55fd9707 with result from task 788
parsl.dataflow.dflow DEBUG: Task 788 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a4b0d233-2d96-4fcd-bc1f-423ecdacbe8d_calibrate_946073_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/a4b0d233-2d96-4fcd-bc1f-423ecdacbe8d_calibrate_946073_33.stderr
parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d52af34353239c920bd3994c0c6097ff with result from task 800
parsl.dataflow.dflow DEBUG: Task 800 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1ffc1a0d-c27b-4754-9652-fda00346189e_calibrate_946073_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/1ffc1a0d-c27b-4754-9652-fda00346189e_calibrate_946073_30.stderr
parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8616fc9ff5079f6eea12a5323e6bd762 with result from task 341
parsl.dataflow.dflow DEBUG: Task 341 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f288bedf-3091-4b46-ba03-cadc3fb3efc9_calibrate_946180_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f288bedf-3091-4b46-ba03-cadc3fb3efc9_calibrate_946180_55.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 71
parsl.dataflow.strategy DEBUG: Executor multi has 71 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5a3557bc6cb9e95a717d21fb57684c5 with result from task 461
parsl.dataflow.dflow DEBUG: Task 461 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8ac11115-0041-4c46-9ecd-2f6361230c96_calibrate_946182_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8ac11115-0041-4c46-9ecd-2f6361230c96_calibrate_946182_9.stderr
parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5215a8a4efec795ab490e10464d33f7 with result from task 1015
parsl.dataflow.dflow DEBUG: Task 1015 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash bbbc760d08c8ad709666dd7a267dd21d
parsl.dataflow.memoization INFO: Task 1016 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1106
parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5a42db39-dea3-4d63-a2c8-9962f29e9c68_calibrate_946182_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5a42db39-dea3-4d63-a2c8-9962f29e9c68_calibrate_946182_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a91f84e7-bf7b-4b80-962f-350fee745422_characterizeImage_946182_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/a91f84e7-bf7b-4b80-962f-350fee745422_characterizeImage_946182_7.stderr
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43bff9c95506d16b52b2d50d74c14ffd with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/3d0fb75f-5dbc-4149-b0db-e6e6939b71d8_calibrate_946181_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/3d0fb75f-5dbc-4149-b0db-e6e6939b71d8_calibrate_946181_36.stderr
parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ddc11f2f92104b1ce5f04c6375a675eb with result from task 977
parsl.dataflow.dflow DEBUG: Task 977 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/447bd682-2688-4a20-bdcf-7b6bb37c7d99_calibrate_946180_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/447bd682-2688-4a20-bdcf-7b6bb37c7d99_calibrate_946180_58.stderr
parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d04c3c04643c0669162c4364fbc9412a with result from task 863
parsl.dataflow.dflow DEBUG: Task 863 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/deaa91ea-8dad-4d4e-99f6-5e4f47f5ad6d_calibrate_946181_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/deaa91ea-8dad-4d4e-99f6-5e4f47f5ad6d_calibrate_946181_23.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 149 try 1 failed
parsl.dataflow.dflow ERROR: Task 149 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a42a3a93f3d5e3c14f3a16b25b9129a with result from task 149
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/20c55bbe-6324-4a18-a232-688fda89198a_calibrate_946074_61.stderr
parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 234455eae36f59871a2d0e0d49e04bba with result from task 791
parsl.dataflow.dflow DEBUG: Task 791 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/45c2ba11-8f52-4167-a5f4-4713c91e8c04_calibrate_946074_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/45c2ba11-8f52-4167-a5f4-4713c91e8c04_calibrate_946074_15.stderr
parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee24ef9360153ec94dc36caa80a652f4 with result from task 593
parsl.dataflow.dflow DEBUG: Task 593 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/64e77cba-6c0f-4797-bee6-c46834052cc1_calibrate_946181_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/64e77cba-6c0f-4797-bee6-c46834052cc1_calibrate_946181_10.stderr
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:16 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:17 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6990964713985c71d5e2cf667b08fb8f with result from task 1004
parsl.dataflow.dflow DEBUG: Task 1004 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5d368f03-d48b-41c0-b0e5-609b58220e40_calibrate_946074_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5d368f03-d48b-41c0-b0e5-609b58220e40_calibrate_946074_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 293 try 0 failed
parsl.dataflow.dflow INFO: Task 293 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash b61921df4ad2d966c34df23f3a1116a1
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 1 launched on executor multi with executor id 1107
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stderr
parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a79af68541756d188c0c758b198d9cb with result from task 1115
parsl.dataflow.dflow DEBUG: Task 1115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/578584c8-694a-4b9f-a8b7-9fc8581bb936_calibrate_946181_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/578584c8-694a-4b9f-a8b7-9fc8581bb936_calibrate_946181_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:18 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 119aac5e083692d9aafd0511596f0e69 with result from task 1049
parsl.dataflow.dflow DEBUG: Task 1049 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c1d21ff8-9e82-4abe-849d-ad4466655add_calibrate_946181_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c1d21ff8-9e82-4abe-849d-ad4466655add_calibrate_946181_11.stderr
parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc28f59c24bdff170937e98533c362c5 with result from task 1012
parsl.dataflow.dflow DEBUG: Task 1012 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash aa78828ffe08f76947d330a034d9872d
parsl.dataflow.memoization INFO: Task 1013 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15230889f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1108
parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d0194c68-f25d-4d56-a0a4-155dcf66f208_calibrate_946181_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d0194c68-f25d-4d56-a0a4-155dcf66f208_calibrate_946181_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6c6b4150-e1ed-4cc6-9bfd-88bb4c6c0d7e_characterizeImage_946181_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946181/6c6b4150-e1ed-4cc6-9bfd-88bb4c6c0d7e_characterizeImage_946181_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.memoization DEBUG: Storing app cache entry fe504f5583d853b56aebbe45d44683c4 with result from task 785
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 785 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7dcdaa85-56bb-4233-a033-7360839b0876_calibrate_946181_59.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7dcdaa85-56bb-4233-a033-7360839b0876_calibrate_946181_59.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:19 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:20 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c52035086ecd4b512626a84bb8065f32 with result from task 551
parsl.dataflow.dflow DEBUG: Task 551 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/52813176-c6d2-4ff2-9a54-be23d48dfd53_calibrate_946181_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/52813176-c6d2-4ff2-9a54-be23d48dfd53_calibrate_946181_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:21 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f3220e8f4255a8df36d231b233919f56 with result from task 752
parsl.dataflow.dflow DEBUG: Task 752 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d2d5dc52-faae-4f89-8f3a-d8aa076d1d23_calibrate_946074_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/d2d5dc52-faae-4f89-8f3a-d8aa076d1d23_calibrate_946074_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d3c899d19f28b22e64cff16c0ad8d56 with result from task 329
parsl.dataflow.dflow DEBUG: Task 329 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6e5c9724-8caf-4c19-8d13-c59a540ca79f_calibrate_946073_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/6e5c9724-8caf-4c19-8d13-c59a540ca79f_calibrate_946073_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:22 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d29724824998bc007b71430226d69ff with result from task 1019
parsl.dataflow.dflow DEBUG: Task 1019 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/fa8c2dd2-592a-4202-8e8b-e01b31fb73cf_calibrate_946073_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/fa8c2dd2-592a-4202-8e8b-e01b31fb73cf_calibrate_946073_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:23 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65300f49175b144aae04909f7a6ef247 with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/32593b3b-dc54-4250-b016-97b880c5d419_calibrate_946182_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/32593b3b-dc54-4250-b016-97b880c5d419_calibrate_946182_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7f51a02c091ea5f5db832551655ad92 with result from task 194
parsl.dataflow.dflow DEBUG: Task 194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/131a090a-2d98-4a2c-9a6b-b9b0fb6ee254_calibrate_946182_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/131a090a-2d98-4a2c-9a6b-b9b0fb6ee254_calibrate_946182_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 191 try 1 failed
parsl.dataflow.dflow ERROR: Task 191 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 89328f12f5079f1230f7439194eefde4 with result from task 191
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d7511761-5971-4b5c-8ef2-302a191bf72d_calibrate_946181_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:24 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24c334f4bf74a4c52813825cba6759d6 with result from task 1037
parsl.dataflow.dflow DEBUG: Task 1037 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7dc7939f-69c5-48c5-8779-e3852fe9aebc_calibrate_946182_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7dc7939f-69c5-48c5-8779-e3852fe9aebc_calibrate_946182_29.stderr
parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 472624522d75345ad33701987c7e4daa with result from task 644
parsl.dataflow.dflow DEBUG: Task 644 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4f43bb63-9d80-4486-b665-79e6a0d32b26_calibrate_946073_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/4f43bb63-9d80-4486-b665-79e6a0d32b26_calibrate_946073_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce9d1a0ff63c2913dd979b87f1f65efa with result from task 827
parsl.dataflow.dflow DEBUG: Task 827 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/cda2399e-80bc-4bb2-8c7c-389f04ee00e2_calibrate_946181_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/cda2399e-80bc-4bb2-8c7c-389f04ee00e2_calibrate_946181_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 064664868ad9bb641419899992849501 with result from task 983
parsl.dataflow.dflow DEBUG: Task 983 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d3835bac-ac8c-4f49-bd6a-01eb7a1192ea_calibrate_946073_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/d3835bac-ac8c-4f49-bd6a-01eb7a1192ea_calibrate_946073_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:25 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff7c1ea8572485f18876d924ba70e94a with result from task 641
parsl.dataflow.dflow DEBUG: Task 641 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2edead1-db90-49fc-98c0-f8e0eb834577_calibrate_946182_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/f2edead1-db90-49fc-98c0-f8e0eb834577_calibrate_946182_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:26 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:27 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:27 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 48
parsl.dataflow.strategy DEBUG: Executor multi has 48 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 48
parsl.dataflow.strategy DEBUG: Executor multi has 48 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c612b129797a5000597fcad4cb05937a with result from task 542
parsl.dataflow.dflow DEBUG: Task 542 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee539c2f-43d9-4e10-af4d-a44a141a035f_calibrate_946072_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee539c2f-43d9-4e10-af4d-a44a141a035f_calibrate_946072_25.stderr
parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca9415e5d64792d3e8566542f8a9c972 with result from task 860
parsl.dataflow.dflow DEBUG: Task 860 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fb98e0f9-6c14-49f2-8c41-df1a50277d77_calibrate_946074_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/fb98e0f9-6c14-49f2-8c41-df1a50277d77_calibrate_946074_24.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 46
parsl.dataflow.strategy DEBUG: Executor multi has 46 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a7b1a3ea26e7a00ebf7e39a2a170357 with result from task 755
parsl.dataflow.dflow DEBUG: Task 755 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/388e51cc-967f-492c-a69a-b80e7401bbaa_calibrate_946181_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/388e51cc-967f-492c-a69a-b80e7401bbaa_calibrate_946181_1.stderr
parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd0ec86ae6f6f730e417a90e131420e0 with result from task 596
parsl.dataflow.dflow DEBUG: Task 596 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2204fe00-3931-4bf5-a12e-bc4ace635dd4_calibrate_946182_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/2204fe00-3931-4bf5-a12e-bc4ace635dd4_calibrate_946182_18.stderr
parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79cc59cc806aef1b64d6b5c6121b2d54 with result from task 646
parsl.dataflow.dflow DEBUG: Task 646 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 45b992775216543dd357279011083214
parsl.dataflow.memoization INFO: Task 647 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 1109
parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3678b9ef-b9b3-48d6-aee7-b0e091c5e6dc_calibrate_946182_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3678b9ef-b9b3-48d6-aee7-b0e091c5e6dc_calibrate_946182_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/d8ebca9a-2b0e-46b6-8fc3-b6627e8fd547_characterizeImage_946182_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/d8ebca9a-2b0e-46b6-8fc3-b6627e8fd547_characterizeImage_946182_32.stderr
parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fa00e6e324e5e34dd0b3165b18f5ca2 with result from task 436
parsl.dataflow.dflow DEBUG: Task 436 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 437 has memoization hash f74fc64fc0a994b5c1583b4462799270
parsl.dataflow.memoization INFO: Task 437 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308a41a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 1110
parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/37899e9f-6b52-4315-9360-8f1129bec6f4_calibrate_946074_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/37899e9f-6b52-4315-9360-8f1129bec6f4_calibrate_946074_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/04392313-25d2-4e7a-86a7-c040b6b2c5e9_characterizeImage_946074_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/04392313-25d2-4e7a-86a7-c040b6b2c5e9_characterizeImage_946074_14.stderr
parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcadf0bd0b9d25e49d78f9a0c1412f00 with result from task 887
parsl.dataflow.dflow DEBUG: Task 887 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0da64d9a-8739-406b-b17c-fab7a4803f90_calibrate_946181_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/0da64d9a-8739-406b-b17c-fab7a4803f90_calibrate_946181_48.stderr
parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3dd739d83abe77fa2726692542e3ab81 with result from task 950
parsl.dataflow.dflow DEBUG: Task 950 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/da8225bf-1f1a-45ed-84bb-a2a4a9efd4e8_calibrate_946181_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/da8225bf-1f1a-45ed-84bb-a2a4a9efd4e8_calibrate_946181_24.stderr
parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4921b434cb89855a8f4f777ce25bb21a with result from task 938
parsl.dataflow.dflow DEBUG: Task 938 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f39d3afe-a6a3-4c3a-9169-e7912bee80ed_calibrate_946181_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/f39d3afe-a6a3-4c3a-9169-e7912bee80ed_calibrate_946181_44.stderr
parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d441d412baeb912f39c8c43df3bc275 with result from task 566
parsl.dataflow.dflow DEBUG: Task 566 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/4fa7167e-d945-4176-848b-fd97f45ff38d_calibrate_946182_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/4fa7167e-d945-4176-848b-fd97f45ff38d_calibrate_946182_15.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 40
parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63ad2eb4ba8509041d7f0a0ff7481274 with result from task 869
parsl.dataflow.dflow DEBUG: Task 869 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7a9bb4f4-3782-44ab-9c32-12c33546ea9a_calibrate_946182_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/7a9bb4f4-3782-44ab-9c32-12c33546ea9a_calibrate_946182_53.stderr
parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02e8650d2cf435f4dfb7bb2a88c50ffd with result from task 905
parsl.dataflow.dflow DEBUG: Task 905 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/7025dd23-17a5-479e-a3f7-0977fded0e0a_calibrate_946074_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/7025dd23-17a5-479e-a3f7-0977fded0e0a_calibrate_946074_12.stderr
parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c8ba6666aed2dc0a48b5ffbfa5db362 with result from task 743
parsl.dataflow.dflow DEBUG: Task 743 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5201bf2b-91c5-4b7b-8574-3912a502b7e8_calibrate_946074_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/5201bf2b-91c5-4b7b-8574-3912a502b7e8_calibrate_946074_18.stderr
parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a8127fb8269199f233600471245c2c41 with result from task 749
parsl.dataflow.dflow DEBUG: Task 749 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e6bf85af-e6c4-4156-bacf-3de785a9fd98_calibrate_946181_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/e6bf85af-e6c4-4156-bacf-3de785a9fd98_calibrate_946181_20.stderr
parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1970c36f22baa9019034fbdf381fa98b with result from task 995
parsl.dataflow.dflow DEBUG: Task 995 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/20a5ccb4-c6d2-4095-9ac7-b158a75bf30e_calibrate_946073_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/20a5ccb4-c6d2-4095-9ac7-b158a75bf30e_calibrate_946073_28.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 35
parsl.dataflow.strategy DEBUG: Executor multi has 35 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfc19122f8f8897e916625b73ac1a6f8 with result from task 416
parsl.dataflow.dflow DEBUG: Task 416 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee78d8ea-e3f6-45b8-a715-13f0148f0bcb_calibrate_946180_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee78d8ea-e3f6-45b8-a715-13f0148f0bcb_calibrate_946180_17.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3284ee5fc71d9daa5b106c0f5697736b with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/90294a33-5006-4795-acdf-a2548c72df76_calibrate_946180_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/90294a33-5006-4795-acdf-a2548c72df76_calibrate_946180_16.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 293 try 1 failed
parsl.dataflow.dflow ERROR: Task 293 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry b61921df4ad2d966c34df23f3a1116a1 with result from task 293
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/65073149-f288-4909-a8fb-b8901beca478_calibrate_946073_61.stderr
parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55352615daddd78196cec6840dc04500 with result from task 872
parsl.dataflow.dflow DEBUG: Task 872 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/de1eb473-1648-45d4-8bc5-02f719322845_calibrate_946182_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/de1eb473-1648-45d4-8bc5-02f719322845_calibrate_946182_56.stderr
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:47 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a489b0eee3a2c9da81199edc61645617 with result from task 1010
parsl.dataflow.dflow DEBUG: Task 1010 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/88919732-ab87-4d2b-a0b0-c5c7b1aae111_calibrate_946180_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/88919732-ab87-4d2b-a0b0-c5c7b1aae111_calibrate_946180_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:48 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:49 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:50 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:51 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f26736d4e600f57e8538ee25e6feea1c with result from task 776
parsl.dataflow.dflow DEBUG: Task 776 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c154fe25-a75f-434d-a8f3-3d5d093c44fe_calibrate_946181_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c154fe25-a75f-434d-a8f3-3d5d093c44fe_calibrate_946181_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:52 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:53 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:54 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:55 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 94f54b437ef48449d911c58e907ad111 with result from task 851
parsl.dataflow.dflow DEBUG: Task 851 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7c439016-9cac-42aa-847d-526ddecae0b3_calibrate_946181_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/7c439016-9cac-42aa-847d-526ddecae0b3_calibrate_946181_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:56 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:57 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:58 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:40:58 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 28
parsl.dataflow.strategy DEBUG: Executor multi has 28 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 28
parsl.dataflow.strategy DEBUG: Executor multi has 28 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37ebe3a33ccc407b227f8631f425147f with result from task 1022
parsl.dataflow.dflow DEBUG: Task 1022 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee3d7c93-5492-4d1b-8862-17c2a0bc2180_calibrate_946072_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/ee3d7c93-5492-4d1b-8862-17c2a0bc2180_calibrate_946072_7.stderr
parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32b029a09c2197c7fc9a47692d77c3fb with result from task 1076
parsl.dataflow.dflow DEBUG: Task 1076 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee60e4e2-8b46-4b97-a3ce-142161cd027b_calibrate_946180_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/ee60e4e2-8b46-4b97-a3ce-142161cd027b_calibrate_946180_6.stderr
parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54c8476111b8572d5b932651e5602e53 with result from task 1034
parsl.dataflow.dflow DEBUG: Task 1034 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/77dd17bf-1237-4331-a825-19b38545dfa9_calibrate_946181_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/77dd17bf-1237-4331-a825-19b38545dfa9_calibrate_946181_15.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 767 try 1 failed
parsl.dataflow.dflow ERROR: Task 767 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 84744e47cb2e5a850b5a996a8e79de66 with result from task 767
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/eea6f138-928b-4dcc-b89d-6325ca4236c4_calibrate_946182_31.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 24
parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e6d3814252f6cc7ab351f2b08420868 with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc2a16b0-0d59-4385-82ba-fc7bb8c19b82_calibrate_946180_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/cc2a16b0-0d59-4385-82ba-fc7bb8c19b82_calibrate_946180_61.stderr
parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3a018d0b95140947f7fe6fb7af0769aa with result from task 740
parsl.dataflow.dflow DEBUG: Task 740 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64e0392e-5cb3-486e-b319-175e2c9f4da3_calibrate_946073_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/64e0392e-5cb3-486e-b319-175e2c9f4da3_calibrate_946073_7.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 22
parsl.dataflow.strategy DEBUG: Executor multi has 22 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f74fc64fc0a994b5c1583b4462799270 with result from task 437
parsl.dataflow.dflow DEBUG: Task 437 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/37899e9f-6b52-4315-9360-8f1129bec6f4_calibrate_946074_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/37899e9f-6b52-4315-9360-8f1129bec6f4_calibrate_946074_14.stderr
parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52378138bf2f96fe87bd474aeba24c3d with result from task 758
parsl.dataflow.dflow DEBUG: Task 758 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24c3c65e-a9aa-49d3-84ce-9945167a49b8_calibrate_946181_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/24c3c65e-a9aa-49d3-84ce-9945167a49b8_calibrate_946181_47.stderr
parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18f3194746e6ba4cf298b2f091147d7b with result from task 1028
parsl.dataflow.dflow DEBUG: Task 1028 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/bb21a9e6-75db-412b-896c-919ee9f5506b_calibrate_946182_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/bb21a9e6-75db-412b-896c-919ee9f5506b_calibrate_946182_37.stderr
parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa78828ffe08f76947d330a034d9872d with result from task 1013
parsl.dataflow.dflow DEBUG: Task 1013 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d0194c68-f25d-4d56-a0a4-155dcf66f208_calibrate_946181_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/d0194c68-f25d-4d56-a0a4-155dcf66f208_calibrate_946181_7.stderr
parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 790dc8ee46b2d68a2aee0f4f0641c42f with result from task 857
parsl.dataflow.dflow DEBUG: Task 857 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/66a4e4a7-be69-45d4-aa86-b825ec2422fb_calibrate_946182_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/66a4e4a7-be69-45d4-aa86-b825ec2422fb_calibrate_946182_26.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 17
parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bbbc760d08c8ad709666dd7a267dd21d with result from task 1016
parsl.dataflow.dflow DEBUG: Task 1016 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5a42db39-dea3-4d63-a2c8-9962f29e9c68_calibrate_946182_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/5a42db39-dea3-4d63-a2c8-9962f29e9c68_calibrate_946182_7.stderr
2024-03-13 21:41:18 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:41:18 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:18 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:19 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry addf2ff53397b0cdfa80e9918cc9a4c5 with result from task 326
parsl.dataflow.dflow DEBUG: Task 326 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b5892b91-2e68-447d-b600-8d474adfe5ef_calibrate_946180_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/b5892b91-2e68-447d-b600-8d474adfe5ef_calibrate_946180_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:20 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f68d28a0e4ff52cd8b5efeefe8b4d7d4 with result from task 998
parsl.dataflow.dflow DEBUG: Task 998 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/cd4f43c0-1a36-45d1-b7f3-d03c41d00162_calibrate_946182_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/cd4f43c0-1a36-45d1-b7f3-d03c41d00162_calibrate_946182_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5254f92ddc4ac52a3d9ae76a01c66c62 with result from task 923
parsl.dataflow.dflow DEBUG: Task 923 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4effc77a-8dd4-409f-bc98-6dc13c41532f_calibrate_946181_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/4effc77a-8dd4-409f-bc98-6dc13c41532f_calibrate_946181_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:21 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:22 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:23 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f74687cb0075684d187a4c29faf16502 with result from task 854
parsl.dataflow.dflow DEBUG: Task 854 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c8e2125f-fa49-4b32-9596-0908f761edf4_calibrate_946181_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/c8e2125f-fa49-4b32-9596-0908f761edf4_calibrate_946181_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:24 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:25 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:26 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:27 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:28 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:29 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:29 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:29 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:29 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:29 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:29 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:30 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:30 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:30 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12
parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12
parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41706e42b16fab43250ccfb9480de2b0 with result from task 253
parsl.dataflow.dflow DEBUG: Task 253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 7859bddfbcd936e5167013c86cb0a508
parsl.dataflow.memoization INFO: Task 254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309ff1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 1111
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/60b455d7-1af3-4ade-bb73-a6c14ecee8a6_calibrate_946180_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/60b455d7-1af3-4ade-bb73-a6c14ecee8a6_calibrate_946180_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c9a78ef7-086c-46b2-8ce5-6830cb310492_characterizeImage_946180_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/c9a78ef7-086c-46b2-8ce5-6830cb310492_characterizeImage_946180_33.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12
parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afb581f232491dd73aa68cbfc3d34c8d with result from task 866
parsl.dataflow.dflow DEBUG: Task 866 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1eaf3599-d8e5-4445-a8ce-a3b9a754f620_calibrate_946181_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946181/1eaf3599-d8e5-4445-a8ce-a3b9a754f620_calibrate_946181_12.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11
parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cb0c4e0c29e5583b2aa87fd98ee8e16 with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 313dea25e567d111ca6573506cfa4c6c
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f89580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 1112
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e0f71367-23fc-4cf7-818c-969370155b90_calibrate_946182_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e0f71367-23fc-4cf7-818c-969370155b90_calibrate_946182_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/4c9174ac-14c6-4f5a-8770-7c7433c1c926_characterizeImage_946182_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/4c9174ac-14c6-4f5a-8770-7c7433c1c926_characterizeImage_946182_61.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11
parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 45b992775216543dd357279011083214 with result from task 647
parsl.dataflow.dflow DEBUG: Task 647 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3678b9ef-b9b3-48d6-aee7-b0e091c5e6dc_calibrate_946182_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/3678b9ef-b9b3-48d6-aee7-b0e091c5e6dc_calibrate_946182_32.stderr
2024-03-13 21:41:50 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:41:50 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:50 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:50 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:50 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:50 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:51 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:52 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:53 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:54 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:55 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:56 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:57 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:58 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:41:59 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:00 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:01 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:01 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:01 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:01 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10
parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10
parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10
parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10
parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10
parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:42:21 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:42:21 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:21 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:21 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7859bddfbcd936e5167013c86cb0a508 with result from task 254
parsl.dataflow.dflow DEBUG: Task 254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/60b455d7-1af3-4ade-bb73-a6c14ecee8a6_calibrate_946180_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/60b455d7-1af3-4ade-bb73-a6c14ecee8a6_calibrate_946180_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:22 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24c000b37834fe74b0912fa15a1e5864 with result from task 1042
parsl.dataflow.dflow DEBUG: Task 1042 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash ce800b2fde24c7dc7d220c2c3db4055c
parsl.dataflow.memoization INFO: Task 1043 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086e9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1113
parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/4bf36dcf-ace0-4745-b664-9e70e9581b12_calibrate_946072_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/4bf36dcf-ace0-4745-b664-9e70e9581b12_calibrate_946072_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d065d8b4-fd29-439f-ac78-c9c5db1b0f1f_characterizeImage_946072_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946072/d065d8b4-fd29-439f-ac78-c9c5db1b0f1f_characterizeImage_946072_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:23 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:24 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:25 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:26 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:27 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:28 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:29 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:30 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:31 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:32 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a71e7052af1ad28ad78a50523c425810 with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 263024c45aec48defb8a6f95c4140932
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309fb36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 1114
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f8ec47b8-56f8-411a-957a-9bf688e6e1e3_calibrate_946180_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f8ec47b8-56f8-411a-957a-9bf688e6e1e3_calibrate_946180_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ec9fb21e-d89e-4498-9da3-42855dfd2f2f_characterizeImage_946180_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/ec9fb21e-d89e-4498-9da3-42855dfd2f2f_characterizeImage_946180_38.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9
parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01a82f898d24c4ae82f93def370e558a with result from task 1078
parsl.dataflow.dflow DEBUG: Task 1078 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 1984fc731e4aa551b4d4511d7fc5247a
parsl.dataflow.memoization INFO: Task 1079 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523086eb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1115
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c391b986-2fda-4560-8001-0113064fb695_calibrate_946074_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c391b986-2fda-4560-8001-0113064fb695_calibrate_946074_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e53092e6-be8c-47b0-8671-babe6e728f16_characterizeImage_946074_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/e53092e6-be8c-47b0-8671-babe6e728f16_characterizeImage_946074_8.stderr
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9b3725bc86aa471b99334c57e5bbe82 with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash f51a84986346977025a6a7010e599ff9
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 1116
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/44d2282a-e050-45f2-85a7-e0c4e0b59ffd_calibrate_946074_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/44d2282a-e050-45f2-85a7-e0c4e0b59ffd_calibrate_946074_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ae255fb8-97c9-4de5-a4b1-72cd37f93775_characterizeImage_946074_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/ae255fb8-97c9-4de5-a4b1-72cd37f93775_characterizeImage_946074_20.stderr
2024-03-13 21:42:52 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:42:52 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce800b2fde24c7dc7d220c2c3db4055c with result from task 1043
parsl.dataflow.dflow DEBUG: Task 1043 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/4bf36dcf-ace0-4745-b664-9e70e9581b12_calibrate_946072_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946072/4bf36dcf-ace0-4745-b664-9e70e9581b12_calibrate_946072_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:53 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:54 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:55 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:56 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:57 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:58 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:42:59 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:00 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:01 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:02 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:03 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:04 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:04 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8
parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8
parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34d8d43cb9d49c433c89d9d945f4d4d2 with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash ee75ae3c9c353cdb85c4641556cc850d
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152309f6e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 1117
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/bb909a17-4ad5-40d7-9b79-616053b43ab0_calibrate_946180_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/bb909a17-4ad5-40d7-9b79-616053b43ab0_calibrate_946180_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e32c4416-778d-4a05-8851-086d17232d1f_characterizeImage_946180_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946180/e32c4416-778d-4a05-8851-086d17232d1f_characterizeImage_946180_25.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8
parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 313dea25e567d111ca6573506cfa4c6c with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e0f71367-23fc-4cf7-818c-969370155b90_calibrate_946182_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/e0f71367-23fc-4cf7-818c-969370155b90_calibrate_946182_61.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7
parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7
parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:24 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:25 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:26 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 263024c45aec48defb8a6f95c4140932 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f8ec47b8-56f8-411a-957a-9bf688e6e1e3_calibrate_946180_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/f8ec47b8-56f8-411a-957a-9bf688e6e1e3_calibrate_946180_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:27 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:28 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f51a84986346977025a6a7010e599ff9 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/44d2282a-e050-45f2-85a7-e0c4e0b59ffd_calibrate_946074_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/44d2282a-e050-45f2-85a7-e0c4e0b59ffd_calibrate_946074_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:29 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:30 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:31 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:32 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:33 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:34 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:35 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:35 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:35 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:35 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:43:55 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:43:55 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:55 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:55 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:55 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1984fc731e4aa551b4d4511d7fc5247a with result from task 1079
parsl.dataflow.dflow DEBUG: Task 1079 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c391b986-2fda-4560-8001-0113064fb695_calibrate_946074_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/c391b986-2fda-4560-8001-0113064fb695_calibrate_946074_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:56 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee75ae3c9c353cdb85c4641556cc850d with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/bb909a17-4ad5-40d7-9b79-616053b43ab0_calibrate_946180_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946180/bb909a17-4ad5-40d7-9b79-616053b43ab0_calibrate_946180_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:57 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:58 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:43:59 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:00 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:01 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:02 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:03 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:04 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:05 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:06 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:06 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:06 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:06 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:44:26 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:44:26 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:26 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:26 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:27 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:28 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:29 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:30 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:31 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:32 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:33 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:34 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:35 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:36 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:37 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:58 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:44:59 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:00 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:01 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af2d3b938b229f7b9f941e3036591d87 with result from task 730
parsl.dataflow.dflow DEBUG: Task 730 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 733607cea8977adde869731253779202
parsl.dataflow.memoization INFO: Task 731 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152308974e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 1118
parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cab6fae-f8b4-4bb2-9d79-bd065c8783b0_calibrate_946182_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cab6fae-f8b4-4bb2-9d79-bd065c8783b0_calibrate_946182_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bfa1bf1f-324c-43c2-aea6-873a082011fd_characterizeImage_946182_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946182/bfa1bf1f-324c-43c2-aea6-873a082011fd_characterizeImage_946182_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:02 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:03 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:04 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:05 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:06 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:07 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:08 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:09 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:29 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:30 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:31 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:32 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:33 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:34 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:35 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:36 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:37 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:38 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:39 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:40 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:40 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:40 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:45:40 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:46:00 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:46:00 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:00 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:00 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:00 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:01 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:02 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:03 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:04 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:05 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:06 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 733607cea8977adde869731253779202 with result from task 731
parsl.dataflow.dflow DEBUG: Task 731 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cab6fae-f8b4-4bb2-9d79-bd065c8783b0_calibrate_946182_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946182/8cab6fae-f8b4-4bb2-9d79-bd065c8783b0_calibrate_946182_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:07 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:08 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:09 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:10 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:11 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:11 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:11 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:11 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:11 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57c0f375fc1eb8fb4231986ff4a78be0 with result from task 793
parsl.dataflow.dflow DEBUG: Task 793 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 0cbad640b85e8cc21fd7ee0b494d5897
parsl.dataflow.memoization INFO: Task 794 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523089c0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1119
parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/283c13c3-5c17-4fa9-b6ab-9dc70d6845a7_calibrate_946073_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/283c13c3-5c17-4fa9-b6ab-9dc70d6845a7_calibrate_946073_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a3bb088-fa68-42f7-9564-c3f97a2a8f2e_characterizeImage_946073_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946073/8a3bb088-fa68-42f7-9564-c3f97a2a8f2e_characterizeImage_946073_40.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa5e961ad7b0a2c3bce7e3f0a4099c62 with result from task 598
parsl.dataflow.dflow DEBUG: Task 598 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 1a6ccfcd097d9648ace9502bd6f22af4
parsl.dataflow.memoization INFO: Task 599 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1523088ec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 1120
parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/a1d82a62-172f-44c5-9af8-1f886a13458f_calibrate_946074_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/a1d82a62-172f-44c5-9af8-1f886a13458f_calibrate_946074_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/84772ee4-b317-40ab-a889-6ea09b32e029_characterizeImage_946074_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/characterizeImage/946074/84772ee4-b317-40ab-a889-6ea09b32e029_characterizeImage_946074_40.stderr
2024-03-13 21:46:31 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:46:31 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:31 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:31 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:32 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:33 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:34 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:35 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:36 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:37 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:38 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:39 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:40 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:41 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:42 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:42 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:42 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:42 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:42 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:46:42 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
2024-03-13 21:47:02 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:47:02 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:03 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:04 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:05 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023666,17023667,17023668,17023669,17023670,17023671,17023672,17023673,17023674,17023675,17023676,17023677,17023678,17023679,17023680,17023681,17023682,17023683,17023684,17023685,17023686,17023687,17023688,17023689,17023690,17023691,17023692,17023693,17023694,17023695'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023689 R
17023690 R
17023691 R
17023692 R
17023693 R
17023694 R
17023695 R
17023685 R
17023686 R
17023687 R
17023688 R
17023684 R
17023681 R
17023682 R
17023683 R
17023679 R
17023680 R
17023678 R
17023675 R
17023676 R
17023677 R
17023671 R
17023672 R
17023673 R
17023674 R
17023666 R
17023667 R
17023668 R
17023669 R
17023670 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023689 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023690 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023691 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023692 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023693 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023694 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023695 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023685 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023686 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023687 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023688 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023684 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023681 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023682 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023683 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023679 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023680 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023678 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023675 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023676 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023677 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023671 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023672 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023673 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023674 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023666 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023667 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023668 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023669 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023670 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:06 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023696,17023697,17023698,17023699,17023700'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023697 R
17023698 R
17023699 R
17023700 R
17023696 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023697 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023698 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023699 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023700 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023696 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:07 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a6ccfcd097d9648ace9502bd6f22af4 with result from task 599
parsl.dataflow.dflow DEBUG: Task 599 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/a1d82a62-172f-44c5-9af8-1f886a13458f_calibrate_946074_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946074/a1d82a62-172f-44c5-9af8-1f886a13458f_calibrate_946074_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cbad640b85e8cc21fd7ee0b494d5897 with result from task 794
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1098
parsl.dataflow.dflow INFO: Tasks in state States.failed: 11
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 7
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: Task 794 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/283c13c3-5c17-4fa9-b6ab-9dc70d6845a7_calibrate_946073_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/submit/DEEP/20201016/B1e/science#step1/20240313T213234Z/logs/calibrate/946073/283c13c3-5c17-4fa9-b6ab-9dc70d6845a7_calibrate_946073_40.stderr
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:08 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023701,17023702,17023703,17023704,17023705,17023706,17023707,17023708,17023709,17023710,17023711,17023712,17023713,17023715,17023716,17023717,17023718,17023719,17023720,17023721'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023711 R
17023712 R
17023713 R
17023715 R
17023716 R
17023717 R
17023718 R
17023719 R
17023720 R
17023721 R
17023701 R
17023702 R
17023703 R
17023704 R
17023705 R
17023706 R
17023707 R
17023708 R
17023709 R
17023710 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023711 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023712 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023713 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023715 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023716 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023717 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023718 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023719 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023720 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023721 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023701 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023702 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023703 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023704 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023705 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023706 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023707 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023708 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023709 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023710 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:09 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:10 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:11 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:12 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
2024-03-13 21:47:13 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17023722,17023723,17023724,17023725,17023726,17023727,17023728,17023729,17023730,17023731,17023732,17023733,17023734,17023735,17023736,17023737,17023738,17023739,17023740,17023741,17023742,17023743,17023744,17023745,17023746,17023747,17023748,17023749,17023750,17023751,17023752,17023753,17023754,17023755,17023756,17023757,17023758'
parsl.providers.slurm.slurm DEBUG: squeue returned 17023750 R
17023751 R
17023752 R
17023753 R
17023754 R
17023755 R
17023756 R
17023757 R
17023758 R
17023740 R
17023741 R
17023742 R
17023743 R
17023744 R
17023745 R
17023746 R
17023747 R
17023748 R
17023749 R
17023730 R
17023731 R
17023732 R
17023733 R
17023734 R
17023735 R
17023736 R
17023737 R
17023738 R
17023739 R
17023722 R
17023723 R
17023724 R
17023725 R
17023726 R
17023727 R
17023728 R
17023729 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17023750 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023751 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023752 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023753 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023754 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023755 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023756 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023757 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023758 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023740 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023741 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023742 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023743 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023744 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023745 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023746 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023747 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023748 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023749 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023730 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023731 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023732 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023733 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023734 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023735 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023736 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023737 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023738 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023739 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023722 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023723 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023724 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023725 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023726 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023727 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023728 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17023729 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 0
parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 93/0 running/pending blocks, and 353 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks
parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (93) than minimum blocks (0)
parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in
parsl.dataflow.strategy DEBUG: Idle time 3.337860107421875e-06s is less than max_idletime 120.0s for executor multi; not scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23240262967760
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a56c93ef20c2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a56c93ef20c2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a8e904c5228
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a8e904c5228
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 737febc3ccba
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 737febc3ccba
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3fee91c859b1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3fee91c859b1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 947a81375bc8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 947a81375bc8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a31d2c6fea6d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a31d2c6fea6d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b9b56accf64f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b9b56accf64f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d78e6d4eca1d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d78e6d4eca1d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04e16ee98977
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04e16ee98977
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00b60ea8f3ec
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00b60ea8f3ec
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d5f19525e05
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d5f19525e05
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bc27c31cfb62
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bc27c31cfb62
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a5fbce29357
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a5fbce29357
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5ab194265fbb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5ab194265fbb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 403586d8d6eb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 403586d8d6eb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 774b1c2f70ed
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 774b1c2f70ed
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00553b4aced5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00553b4aced5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3dde12cf949c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3dde12cf949c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6280cb5b3aab
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6280cb5b3aab
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ca94b72d67d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ca94b72d67d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 679d2858d60d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 679d2858d60d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f786d255a902
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f786d255a902
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2e9c8776045
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2e9c8776045
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 192efee5f833
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 192efee5f833
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c56ec80c215
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c56ec80c215
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91449ea3f325
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91449ea3f325
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47ebaa394a34
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47ebaa394a34
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 966f175d80f7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 966f175d80f7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 54e5f65a98a5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 54e5f65a98a5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 59af6a67c945
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 59af6a67c945
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da25b44cd671
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da25b44cd671
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 495eec386622
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 495eec386622
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61c4e1acaf9d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61c4e1acaf9d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29ada9ba000e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29ada9ba000e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 041430a605d5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 041430a605d5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7724e47f2426
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7724e47f2426
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd9a9630ab7d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd9a9630ab7d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62ade473abf6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62ade473abf6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b3d1c098ac22
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b3d1c098ac22
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e7e38652a6a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e7e38652a6a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1976d0036eb1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1976d0036eb1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b65a68eeb18f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b65a68eeb18f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a07007e15467
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a07007e15467
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9f3d4e98d2b5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9f3d4e98d2b5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c14c5cf15fc0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c14c5cf15fc0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: facb807a33c4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: facb807a33c4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2604a02e58ff
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2604a02e58ff
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e7c605ae4345
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e7c605ae4345
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c525fcde6ebf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c525fcde6ebf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff0691b4c900
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff0691b4c900
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f2df58bb0448
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f2df58bb0448
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa45d86b826e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa45d86b826e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9de7ea23e4f5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9de7ea23e4f5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 900fe6222052
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 900fe6222052
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ed76a73d5a9c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ed76a73d5a9c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d5883fc67c5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d5883fc67c5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4a890e9c336
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4a890e9c336
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d733ac4f7fca
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d733ac4f7fca
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9215beb9f21e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9215beb9f21e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87d9008a2e68
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87d9008a2e68
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09df43ea664f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09df43ea664f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 518fc9daea93
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 518fc9daea93
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a4ec5da57f0a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a4ec5da57f0a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a5d3059ea53
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a5d3059ea53
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c33dd5b90c33
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c33dd5b90c33
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c460611ecbef
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c460611ecbef
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10ce968cdd60
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10ce968cdd60
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 45e762b5b93b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 45e762b5b93b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11b6edad5f23
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11b6edad5f23
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be46887e26a4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be46887e26a4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba049740cd43
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba049740cd43
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cc00ae8208aa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cc00ae8208aa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d22b3d1acaf0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d22b3d1acaf0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: efb98e327161
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: efb98e327161
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 42136735e899
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 42136735e899
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d3447df7c899
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d3447df7c899
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc92c26044cf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc92c26044cf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a66b0a2e1f02
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a66b0a2e1f02
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7177916c4085
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7177916c4085
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e48cb8fff2f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e48cb8fff2f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d364e704d7a2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d364e704d7a2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f6ac3bb4d62b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f6ac3bb4d62b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 06a4b4a2256f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 06a4b4a2256f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d6be414a935
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d6be414a935
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60d923adc201
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60d923adc201
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 255b788ac5ba
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 255b788ac5ba
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7d2edf0c850
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7d2edf0c850
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 58044bca61b7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 58044bca61b7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 021b233d7b32
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 021b233d7b32
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c25d313a4e7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c25d313a4e7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b987a112e18
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b987a112e18
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fdc8721998c3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fdc8721998c3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aab5ee908185
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aab5ee908185
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023703 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023703 on provider escience_a40
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023671 on provider astro
proc_lsst.multi INFO: cancelling 17023671 on provider astro
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023727 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023727 on provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023732 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023732 on provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023740 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023740 on provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023736 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023736 on provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023678 on provider astro
proc_lsst.multi INFO: cancelling 17023678 on provider astro
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023753 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023753 on provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023683 on provider astro
proc_lsst.multi INFO: cancelling 17023683 on provider astro
2024-03-13 21:47:14 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
2024-03-13 21:47:14 proc_lsst.multi:201 [INFO]  cancelling 17023723 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023723 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023751 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023751 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023750 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023750 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023749 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023749 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023730 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023730 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023722 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023722 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023699 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023699 on provider escience_rtx6k
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023738 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023738 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023666 on provider astro
proc_lsst.multi INFO: cancelling 17023666 on provider astro
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023696 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023696 on provider escience_rtx6k
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:201 [INFO]  cancelling 17023752 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023752 on provider checkpoint
2024-03-13 21:47:15 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023741 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023741 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023757 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023757 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023720 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023720 on provider escience_a40
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023724 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023724 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023745 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023745 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023737 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023737 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023733 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023733 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023744 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023744 on provider checkpoint
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023702 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023702 on provider escience_a40
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023673 on provider astro
proc_lsst.multi INFO: cancelling 17023673 on provider astro
2024-03-13 21:47:16 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
2024-03-13 21:47:16 proc_lsst.multi:201 [INFO]  cancelling 17023680 on provider astro
proc_lsst.multi INFO: cancelling 17023680 on provider astro
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023704 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023704 on provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023717 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023717 on provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023710 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023710 on provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023746 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023746 on provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023734 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023734 on provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023739 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023739 on provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023693 on provider astro
proc_lsst.multi INFO: cancelling 17023693 on provider astro
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023668 on provider astro
proc_lsst.multi INFO: cancelling 17023668 on provider astro
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023712 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023712 on provider escience_a40
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023742 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023742 on provider checkpoint
2024-03-13 21:47:17 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
2024-03-13 21:47:17 proc_lsst.multi:201 [INFO]  cancelling 17023688 on provider astro
proc_lsst.multi INFO: cancelling 17023688 on provider astro
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023758 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023758 on provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023728 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023728 on provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023756 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023756 on provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023698 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023698 on provider escience_rtx6k
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023747 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023747 on provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023716 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023716 on provider escience_a40
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023755 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023755 on provider checkpoint
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023686 on provider astro
proc_lsst.multi INFO: cancelling 17023686 on provider astro
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023681 on provider astro
proc_lsst.multi INFO: cancelling 17023681 on provider astro
2024-03-13 21:47:18 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
2024-03-13 21:47:18 proc_lsst.multi:201 [INFO]  cancelling 17023667 on provider astro
proc_lsst.multi INFO: cancelling 17023667 on provider astro
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023731 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023731 on provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023685 on provider astro
proc_lsst.multi INFO: cancelling 17023685 on provider astro
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023718 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023718 on provider escience_a40
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023754 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023754 on provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023729 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023729 on provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023748 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023748 on provider checkpoint
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023690 on provider astro
proc_lsst.multi INFO: cancelling 17023690 on provider astro
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023715 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023715 on provider escience_a40
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023713 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023713 on provider escience_a40
2024-03-13 21:47:19 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
2024-03-13 21:47:19 proc_lsst.multi:201 [INFO]  cancelling 17023672 on provider astro
proc_lsst.multi INFO: cancelling 17023672 on provider astro
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023677 on provider astro
proc_lsst.multi INFO: cancelling 17023677 on provider astro
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023735 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023735 on provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 12374 on provider local
proc_lsst.multi INFO: cancelling 12374 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 12374
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023743 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023743 on provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023694 on provider astro
proc_lsst.multi INFO: cancelling 17023694 on provider astro
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023726 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023726 on provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023670 on provider astro
proc_lsst.multi INFO: cancelling 17023670 on provider astro
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023679 on provider astro
proc_lsst.multi INFO: cancelling 17023679 on provider astro
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023725 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023725 on provider checkpoint
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023721 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023721 on provider escience_a40
2024-03-13 21:47:20 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
2024-03-13 21:47:20 proc_lsst.multi:201 [INFO]  cancelling 17023719 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023719 on provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023689 on provider astro
proc_lsst.multi INFO: cancelling 17023689 on provider astro
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023701 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023701 on provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023669 on provider astro
proc_lsst.multi INFO: cancelling 17023669 on provider astro
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023705 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023705 on provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023675 on provider astro
proc_lsst.multi INFO: cancelling 17023675 on provider astro
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023700 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023700 on provider escience_rtx6k
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023709 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023709 on provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023691 on provider astro
proc_lsst.multi INFO: cancelling 17023691 on provider astro
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023707 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023707 on provider escience_a40
2024-03-13 21:47:21 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
2024-03-13 21:47:21 proc_lsst.multi:201 [INFO]  cancelling 17023692 on provider astro
proc_lsst.multi INFO: cancelling 17023692 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023674 on provider astro
proc_lsst.multi INFO: cancelling 17023674 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023697 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023697 on provider escience_rtx6k
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023695 on provider astro
proc_lsst.multi INFO: cancelling 17023695 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023682 on provider astro
proc_lsst.multi INFO: cancelling 17023682 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023687 on provider astro
proc_lsst.multi INFO: cancelling 17023687 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023684 on provider astro
proc_lsst.multi INFO: cancelling 17023684 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023706 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023706 on provider escience_a40
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023708 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023708 on provider escience_a40
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023676 on provider astro
proc_lsst.multi INFO: cancelling 17023676 on provider astro
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023711 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023711 on provider escience_a40
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-13 21:47:22 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-13 21:47:22 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['17023722', '17023723', '17023724', '17023725', '17023726', '17023727', '17023728', '17023729', '17023730', '17023731', '17023732', '17023733', '17023734', '17023735', '17023736', '17023737', '17023738', '17023739', '17023740', '17023741', '17023742', '17023743', '17023744', '17023745', '17023746', '17023747', '17023748', '17023749', '17023750', '17023751', '17023752', '17023753', '17023754', '17023755', '17023756', '17023757', '17023758', '17023666', '17023667', '17023668', '17023669', '17023670', '17023671', '17023672', '17023673', '17023674', '17023675', '17023676', '17023677', '17023678', '17023679', '17023680', '17023681', '17023682', '17023683', '17023684', '17023685', '17023686', '17023687', '17023688', '17023689', '17023690', '17023691', '17023692', '17023693', '17023694', '17023695', '17023701', '17023702', '17023703', '17023704', '17023705', '17023706', '17023707', '17023708', '17023709', '17023710', '17023711', '17023712', '17023713', '17023715', '17023716', '17023717', '17023718', '17023719', '17023720', '17023721', '17023696', '17023697', '17023698', '17023699', '17023700', '12374']
proc_lsst.multi INFO: new jobs since last cancel ['17023722', '17023723', '17023724', '17023725', '17023726', '17023727', '17023728', '17023729', '17023730', '17023731', '17023732', '17023733', '17023734', '17023735', '17023736', '17023737', '17023738', '17023739', '17023740', '17023741', '17023742', '17023743', '17023744', '17023745', '17023746', '17023747', '17023748', '17023749', '17023750', '17023751', '17023752', '17023753', '17023754', '17023755', '17023756', '17023757', '17023758', '17023666', '17023667', '17023668', '17023669', '17023670', '17023671', '17023672', '17023673', '17023674', '17023675', '17023676', '17023677', '17023678', '17023679', '17023680', '17023681', '17023682', '17023683', '17023684', '17023685', '17023686', '17023687', '17023688', '17023689', '17023690', '17023691', '17023692', '17023693', '17023694', '17023695', '17023701', '17023702', '17023703', '17023704', '17023705', '17023706', '17023707', '17023708', '17023709', '17023710', '17023711', '17023712', '17023713', '17023715', '17023716', '17023717', '17023718', '17023719', '17023720', '17023721', '17023696', '17023697', '17023698', '17023699', '17023700', '12374']
2024-03-13 21:47:22 proc_lsst.multi:146 [INFO]  found job 17023722 in provider checkpoint
proc_lsst.multi INFO: found job 17023722 in provider checkpoint
2024-03-13 21:47:22 proc_lsst.multi:201 [INFO]  cancelling 17023722 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023722 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023723 in provider checkpoint
proc_lsst.multi INFO: found job 17023723 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023723 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023723 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023724 in provider checkpoint
proc_lsst.multi INFO: found job 17023724 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023724 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023724 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023725 in provider checkpoint
proc_lsst.multi INFO: found job 17023725 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023725 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023725 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023726 in provider checkpoint
proc_lsst.multi INFO: found job 17023726 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023726 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023726 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023727 in provider checkpoint
proc_lsst.multi INFO: found job 17023727 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023727 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023727 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023728 in provider checkpoint
proc_lsst.multi INFO: found job 17023728 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023728 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023728 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023729 in provider checkpoint
proc_lsst.multi INFO: found job 17023729 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023729 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023729 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023730 in provider checkpoint
proc_lsst.multi INFO: found job 17023730 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023730 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023730 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023731 in provider checkpoint
proc_lsst.multi INFO: found job 17023731 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023731 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023731 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023732 in provider checkpoint
proc_lsst.multi INFO: found job 17023732 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023732 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023732 on provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:146 [INFO]  found job 17023733 in provider checkpoint
proc_lsst.multi INFO: found job 17023733 in provider checkpoint
2024-03-13 21:47:23 proc_lsst.multi:201 [INFO]  cancelling 17023733 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023733 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023734 in provider checkpoint
proc_lsst.multi INFO: found job 17023734 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023734 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023734 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023735 in provider checkpoint
proc_lsst.multi INFO: found job 17023735 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023735 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023735 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023736 in provider checkpoint
proc_lsst.multi INFO: found job 17023736 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023736 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023736 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023737 in provider checkpoint
proc_lsst.multi INFO: found job 17023737 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023737 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023737 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023738 in provider checkpoint
proc_lsst.multi INFO: found job 17023738 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023738 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023738 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023739 in provider checkpoint
proc_lsst.multi INFO: found job 17023739 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023739 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023739 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023740 in provider checkpoint
proc_lsst.multi INFO: found job 17023740 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023740 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023740 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023741 in provider checkpoint
proc_lsst.multi INFO: found job 17023741 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023741 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023741 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023742 in provider checkpoint
proc_lsst.multi INFO: found job 17023742 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023742 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023742 on provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:146 [INFO]  found job 17023743 in provider checkpoint
proc_lsst.multi INFO: found job 17023743 in provider checkpoint
2024-03-13 21:47:24 proc_lsst.multi:201 [INFO]  cancelling 17023743 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023743 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023744 in provider checkpoint
proc_lsst.multi INFO: found job 17023744 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023744 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023744 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023745 in provider checkpoint
proc_lsst.multi INFO: found job 17023745 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023745 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023745 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023746 in provider checkpoint
proc_lsst.multi INFO: found job 17023746 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023746 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023746 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023747 in provider checkpoint
proc_lsst.multi INFO: found job 17023747 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023747 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023747 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023748 in provider checkpoint
proc_lsst.multi INFO: found job 17023748 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023748 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023748 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023749 in provider checkpoint
proc_lsst.multi INFO: found job 17023749 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023749 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023749 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023750 in provider checkpoint
proc_lsst.multi INFO: found job 17023750 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023750 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023750 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023751 in provider checkpoint
proc_lsst.multi INFO: found job 17023751 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023751 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023751 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023752 in provider checkpoint
proc_lsst.multi INFO: found job 17023752 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023752 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023752 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023753 in provider checkpoint
proc_lsst.multi INFO: found job 17023753 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023753 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023753 on provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:146 [INFO]  found job 17023754 in provider checkpoint
proc_lsst.multi INFO: found job 17023754 in provider checkpoint
2024-03-13 21:47:25 proc_lsst.multi:201 [INFO]  cancelling 17023754 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023754 on provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023755 in provider checkpoint
proc_lsst.multi INFO: found job 17023755 in provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023755 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023755 on provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023756 in provider checkpoint
proc_lsst.multi INFO: found job 17023756 in provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023756 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023756 on provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023757 in provider checkpoint
proc_lsst.multi INFO: found job 17023757 in provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023757 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023757 on provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023758 in provider checkpoint
proc_lsst.multi INFO: found job 17023758 in provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023758 on provider checkpoint
proc_lsst.multi INFO: cancelling 17023758 on provider checkpoint
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023666 in provider astro
proc_lsst.multi INFO: found job 17023666 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023666 on provider astro
proc_lsst.multi INFO: cancelling 17023666 on provider astro
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023667 in provider astro
proc_lsst.multi INFO: found job 17023667 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023667 on provider astro
proc_lsst.multi INFO: cancelling 17023667 on provider astro
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023668 in provider astro
proc_lsst.multi INFO: found job 17023668 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023668 on provider astro
proc_lsst.multi INFO: cancelling 17023668 on provider astro
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023669 in provider astro
proc_lsst.multi INFO: found job 17023669 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023669 on provider astro
proc_lsst.multi INFO: cancelling 17023669 on provider astro
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023670 in provider astro
proc_lsst.multi INFO: found job 17023670 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023670 on provider astro
proc_lsst.multi INFO: cancelling 17023670 on provider astro
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023671 in provider astro
proc_lsst.multi INFO: found job 17023671 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023671 on provider astro
proc_lsst.multi INFO: cancelling 17023671 on provider astro
2024-03-13 21:47:26 proc_lsst.multi:146 [INFO]  found job 17023672 in provider astro
proc_lsst.multi INFO: found job 17023672 in provider astro
2024-03-13 21:47:26 proc_lsst.multi:201 [INFO]  cancelling 17023672 on provider astro
proc_lsst.multi INFO: cancelling 17023672 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023673 in provider astro
proc_lsst.multi INFO: found job 17023673 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023673 on provider astro
proc_lsst.multi INFO: cancelling 17023673 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023674 in provider astro
proc_lsst.multi INFO: found job 17023674 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023674 on provider astro
proc_lsst.multi INFO: cancelling 17023674 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023675 in provider astro
proc_lsst.multi INFO: found job 17023675 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023675 on provider astro
proc_lsst.multi INFO: cancelling 17023675 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023676 in provider astro
proc_lsst.multi INFO: found job 17023676 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023676 on provider astro
proc_lsst.multi INFO: cancelling 17023676 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023677 in provider astro
proc_lsst.multi INFO: found job 17023677 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023677 on provider astro
proc_lsst.multi INFO: cancelling 17023677 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023678 in provider astro
proc_lsst.multi INFO: found job 17023678 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023678 on provider astro
proc_lsst.multi INFO: cancelling 17023678 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023679 in provider astro
proc_lsst.multi INFO: found job 17023679 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023679 on provider astro
proc_lsst.multi INFO: cancelling 17023679 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023680 in provider astro
proc_lsst.multi INFO: found job 17023680 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023680 on provider astro
proc_lsst.multi INFO: cancelling 17023680 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023681 in provider astro
proc_lsst.multi INFO: found job 17023681 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023681 on provider astro
proc_lsst.multi INFO: cancelling 17023681 on provider astro
2024-03-13 21:47:27 proc_lsst.multi:146 [INFO]  found job 17023682 in provider astro
proc_lsst.multi INFO: found job 17023682 in provider astro
2024-03-13 21:47:27 proc_lsst.multi:201 [INFO]  cancelling 17023682 on provider astro
proc_lsst.multi INFO: cancelling 17023682 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023683 in provider astro
proc_lsst.multi INFO: found job 17023683 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023683 on provider astro
proc_lsst.multi INFO: cancelling 17023683 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023684 in provider astro
proc_lsst.multi INFO: found job 17023684 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023684 on provider astro
proc_lsst.multi INFO: cancelling 17023684 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023685 in provider astro
proc_lsst.multi INFO: found job 17023685 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023685 on provider astro
proc_lsst.multi INFO: cancelling 17023685 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023686 in provider astro
proc_lsst.multi INFO: found job 17023686 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023686 on provider astro
proc_lsst.multi INFO: cancelling 17023686 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023687 in provider astro
proc_lsst.multi INFO: found job 17023687 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023687 on provider astro
proc_lsst.multi INFO: cancelling 17023687 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023688 in provider astro
proc_lsst.multi INFO: found job 17023688 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023688 on provider astro
proc_lsst.multi INFO: cancelling 17023688 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023689 in provider astro
proc_lsst.multi INFO: found job 17023689 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023689 on provider astro
proc_lsst.multi INFO: cancelling 17023689 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023690 in provider astro
proc_lsst.multi INFO: found job 17023690 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023690 on provider astro
proc_lsst.multi INFO: cancelling 17023690 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023691 in provider astro
proc_lsst.multi INFO: found job 17023691 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023691 on provider astro
proc_lsst.multi INFO: cancelling 17023691 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023692 in provider astro
proc_lsst.multi INFO: found job 17023692 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023692 on provider astro
proc_lsst.multi INFO: cancelling 17023692 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023693 in provider astro
proc_lsst.multi INFO: found job 17023693 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023693 on provider astro
proc_lsst.multi INFO: cancelling 17023693 on provider astro
2024-03-13 21:47:28 proc_lsst.multi:146 [INFO]  found job 17023694 in provider astro
proc_lsst.multi INFO: found job 17023694 in provider astro
2024-03-13 21:47:28 proc_lsst.multi:201 [INFO]  cancelling 17023694 on provider astro
proc_lsst.multi INFO: cancelling 17023694 on provider astro
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023695 in provider astro
proc_lsst.multi INFO: found job 17023695 in provider astro
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023695 on provider astro
proc_lsst.multi INFO: cancelling 17023695 on provider astro
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023701 in provider escience_a40
proc_lsst.multi INFO: found job 17023701 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023701 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023701 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023702 in provider escience_a40
proc_lsst.multi INFO: found job 17023702 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023702 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023702 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023703 in provider escience_a40
proc_lsst.multi INFO: found job 17023703 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023703 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023703 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023704 in provider escience_a40
proc_lsst.multi INFO: found job 17023704 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023704 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023704 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023705 in provider escience_a40
proc_lsst.multi INFO: found job 17023705 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023705 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023705 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023706 in provider escience_a40
proc_lsst.multi INFO: found job 17023706 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023706 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023706 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023707 in provider escience_a40
proc_lsst.multi INFO: found job 17023707 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023707 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023707 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023708 in provider escience_a40
proc_lsst.multi INFO: found job 17023708 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023708 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023708 on provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:146 [INFO]  found job 17023709 in provider escience_a40
proc_lsst.multi INFO: found job 17023709 in provider escience_a40
2024-03-13 21:47:29 proc_lsst.multi:201 [INFO]  cancelling 17023709 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023709 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023710 in provider escience_a40
proc_lsst.multi INFO: found job 17023710 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023710 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023710 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023711 in provider escience_a40
proc_lsst.multi INFO: found job 17023711 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023711 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023711 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023712 in provider escience_a40
proc_lsst.multi INFO: found job 17023712 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023712 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023712 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023713 in provider escience_a40
proc_lsst.multi INFO: found job 17023713 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023713 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023713 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023715 in provider escience_a40
proc_lsst.multi INFO: found job 17023715 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023715 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023715 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023716 in provider escience_a40
proc_lsst.multi INFO: found job 17023716 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023716 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023716 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023717 in provider escience_a40
proc_lsst.multi INFO: found job 17023717 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023717 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023717 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023718 in provider escience_a40
proc_lsst.multi INFO: found job 17023718 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023718 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023718 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023719 in provider escience_a40
proc_lsst.multi INFO: found job 17023719 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023719 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023719 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023720 in provider escience_a40
proc_lsst.multi INFO: found job 17023720 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023720 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023720 on provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:146 [INFO]  found job 17023721 in provider escience_a40
proc_lsst.multi INFO: found job 17023721 in provider escience_a40
2024-03-13 21:47:30 proc_lsst.multi:201 [INFO]  cancelling 17023721 on provider escience_a40
proc_lsst.multi INFO: cancelling 17023721 on provider escience_a40
2024-03-13 21:47:31 proc_lsst.multi:146 [INFO]  found job 17023696 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023696 in provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:201 [INFO]  cancelling 17023696 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023696 on provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:146 [INFO]  found job 17023697 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023697 in provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:201 [INFO]  cancelling 17023697 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023697 on provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:146 [INFO]  found job 17023698 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023698 in provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:201 [INFO]  cancelling 17023698 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023698 on provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:146 [INFO]  found job 17023699 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023699 in provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:201 [INFO]  cancelling 17023699 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023699 on provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:146 [INFO]  found job 17023700 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17023700 in provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:201 [INFO]  cancelling 17023700 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17023700 on provider escience_rtx6k
2024-03-13 21:47:31 proc_lsst.multi:146 [INFO]  found job 12374 in provider local
proc_lsst.multi INFO: found job 12374 in provider local
2024-03-13 21:47:31 proc_lsst.multi:201 [INFO]  cancelling 12374 on provider local
proc_lsst.multi INFO: cancelling 12374 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 12374
parsl.providers.local.local WARNING: Failed to kill PID: 12374 and child processes on local
2024-03-13 21:47:32 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 845.7048 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201016_B1e_science#step1_20240313T213234Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 845.7110 seconds; current memory usage: 0.388 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 886.1853 seconds; current memory usage: 0.388 Gibyte, delta: 0.251 Gibyte, peak delta: 0.251 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.388 Gibyte (main), 0.388 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20201016_B1e_science#step1_20240313T213234Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	14m48.555s
user	3m19.728s
sys	2m2.911s